Jump to content

Numeric Formatting


Jason D.

Recommended Posts

The default formatting of %#g only provides NI's legacy default of six digits of precision. That means that most doubles are pretty much cast to singles when they go through EasyXML. Also, digits of precision is inferior to significant digits, because with the former, the number of digits preserved is dependent on the magnitude of the number.

For example:

100.12345678 => 100.123457

and with

1.0012345678 => 1.001235

It's floating point for 's sake. :) The whole idea is that magnitude doesn't affect precision.

 

Is there any way you can put %^#_14g into the numeric formatting? You could also use %^#_7g for singles, but that's icing on the cake. The NI Help recommends using only 13 and 6 digits, because they don't want you to use the last uncertain digit. However, the application (including EasyXML) should preserve that digit so that precision is maintained as long as possible throughout computations.

 

I also like the caret (^=engineering notation) because it makes the exponential numbers slightly more readable, and some measure of human readability is part of what XML is about.

Link to comment
Share on other sites

Hi Jason,

 

Thanks for the great feedback.

 

First, we're considering adding a feature that will allow you to override the default formatting for floating points. I agree that you often need more control over precision/significant figures.

 

Second, I agree that using significant digits is superior to precision and that it would be helpful to include more digits by default. We'll certainly consider changing the default in a future release.

 

Thanks,

 

-Jim

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.