Questions tagged [decimal-precision]
19 questions
4
votes
1 answer
Pandas transpose resets decimal rounding
I am defining decimal points carefully by column in a large results table, but when I transpose it so it can fit on a page in my report (using .to_latex), pandas sets the decimals to whatever the largest number is.
Is it possible to keep my decimal…
nicolejane33
- 422
- 1
- 5
- 13
3
votes
2 answers
Count the scale of a given decimal
How can I count the scale of a given decimal in Powershell?
$a = 0.0001
$b = 0.000001
Casting $a to a string and returning $a.Length gives a result of 6...I need 4.
I thought there'd be a decimal or math function but I haven't found it and messing…
felixmc
- 516
- 1
- 4
- 19
2
votes
2 answers
Find a decimal with a precision of 2 using yes pattern|no pattern
Can someone please point out the mistake I am making, why 123 is not matching here?
def is_decimal(num):
import re
pattern = re.compile(r"(\b\d+\.)(?(1)\d{1,2}\b|\d+)")
result = pattern.search(num)
return…
Rahul verma
- 29
- 7
2
votes
0 answers
Decimal Precision when transferring Data between SQL Server and MySQL
I set up a linked server within Microsoft SQL Server. The linked server points at a MySQL database. I am trying to select data from SQL Server and insert it into the MySQL database.
I have the following query transferring pricing information from…
eclaire211
- 101
- 2
- 12
1
vote
2 answers
How to store a decimal value up to 2 decimal places in another variable in C?
Suppose a = 3.91900007534.
In C language, I want to store the value of the variable a in another variable, r, up to two decimal places, such that r = 3.92.
Note that I don't want to print the value up to two decimal places, I just want to store the…
Imtiaz
- 25
- 2
- 7
1
vote
1 answer
Decimal precision in character field lost on input
I am trying to convert a character variable containing a number into a numeric variable without losing decimal precision. Here is an example with output:
data _null_;
format res 20.17;
res = input("54.78566360473633",20.17);
put res;
run;
and the…
Rookatu
- 1,487
- 3
- 21
- 50
1
vote
2 answers
Odoo decimal precision based on Unit Of Measure
Anybody an idea how to set the decimal precision of a product based on the unit of measure it is sold it.
For example product X sold in Kg has a decimal precision of 3 => Qty: 10.000Kg
Product Y sold in Unit(s) had a decimal precision of 0 -> Qty:…
Jesse
- 727
- 13
- 44
1
vote
1 answer
SQL calculation output precision
Could someone explain the behaviour of this query:
SELECT
0.1 + 0.01,
'',
(CAST(0.1 AS numeric(18,1)))+(CAST(0.01 AS numeric (18,2))),
0.1 + (CAST(0.01 AS numeric (18,2))),
(CAST(0.1 AS numeric(18,1)))+ 0.01,
'',
…
Brondahl
- 7,402
- 5
- 45
- 74
0
votes
1 answer
Division (Transact-SQL) and Data Type Precedence with a decimal(38, 10) data type denominator
According to the documentation / (Division) (Transact-SQL) the result type is related to the Data Type Precedence, that said, in the following scenario, is there any way to receive the result type as a decimal(38, 10) ?
The calculation result seems…
marco13_457
- 1
- 1
0
votes
0 answers
Unexpected CAST decimal Value Objects to decimal(18,2)
I have a value object named "WithdrawFee"
public Fee WithdrawFee { get; private set; }
and the Fee value object is :
public class Fee : ValueObject
{
public decimal Value { get; private set; }
public static Fee FromDecimal(decimal…
Moradof
- 453
- 3
- 14
0
votes
1 answer
double overloaded sin() and cos() do not maintain 15-digit decimal precision
Using this link as a guide, https://www.geeksforgeeks.org/difference-float-double-c-cpp/#:~:text=double%20is%20a%2064%20bit,15%20decimal%20digits%20of%20precision. double is a 64 bit IEEE 754 double precision Floating Point Number (1 bit for the…
asjhdbashjdbasjhdbhjb
- 68
- 2
- 10
0
votes
1 answer
Why doesn't decimal.getcontext().prec=3 work for decimal.Decimal(1.234)
Why doesn't decimal.getcontext().prec for decimal.Decimal(x)? It gives a long decimal value:
>>>decimal.getcontext().prec = 3
>>>decimal.Decimal(2.345)
Decimal('2.345000000000000195399252334027551114559173583984375')
Prashant Singh
- 156
- 1
- 5
0
votes
2 answers
double precision on linux using fpu_control.h
i am trying to port a particular piece of code from solaris to Linux. During the process i found that the precision on linux is different and it is in extended precision and we need to set it to double precision explicitly. To achieve this found…
girish s
- 21
- 3
0
votes
3 answers
How can I get my calculated SQL Server percentage value to only display two decimals?
I've got this TSQL:
CREATE TABLE #TEMPCOMBINED(
DESCRIPTION VARCHAR(200),
WEEK1USAGE DECIMAL(18,2),
WEEK2USAGE DECIMAL(18,2),
USAGEVARIANCE AS WEEK2USAGE - WEEK1USAGE,
WEEK1PRICE DECIMAL(18,2),
WEEK2PRICE DECIMAL(18,2),
PRICEVARIANCE…
B. Clay Shannon-B. Crow Raven
- 8,547
- 144
- 472
- 862
0
votes
1 answer
Dojo number rounding AND decimal places for NumberTextBox / NumberSpinner
I would like to have a NumberSpinner and/or NumberTextBox be able to do two things:
Display a value with a precision of 1 decimal place.
Round those values to the .5
I would also like it if the rounding occured while typing or at the minimum when…
hungerstar
- 21,206
- 6
- 50
- 59