Jump to content
Sign in to follow this  
Jason D.

Numeric Formatting

Recommended Posts

The default formatting of %#g only provides NI's legacy default of six digits of precision. That means that most doubles are pretty much cast to singles when they go through EasyXML. Also, digits of precision is inferior to significant digits, because with the former, the number of digits preserved is dependent on the magnitude of the number.

For example:

100.12345678 => 100.123457

and with

1.0012345678 => 1.001235

It's floating point for 's sake. :) The whole idea is that magnitude doesn't affect precision.

 

Is there any way you can put %^#_14g into the numeric formatting? You could also use %^#_7g for singles, but that's icing on the cake. The NI Help recommends using only 13 and 6 digits, because they don't want you to use the last uncertain digit. However, the application (including EasyXML) should preserve that digit so that precision is maintained as long as possible throughout computations.

 

I also like the caret (^=engineering notation) because it makes the exponential numbers slightly more readable, and some measure of human readability is part of what XML is about.

Share this post


Link to post
Share on other sites

Hi Jason,

 

Thanks for the great feedback.

 

First, we're considering adding a feature that will allow you to override the default formatting for floating points. I agree that you often need more control over precision/significant figures.

 

Second, I agree that using significant digits is superior to precision and that it would be helpful to include more digits by default. We'll certainly consider changing the default in a future release.

 

Thanks,

 

-Jim

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.