Flatness in the USA

Many mechanical interfaces depend on the control of “flatness”.  But what do we really mean by “flatness”?  This is a question that is being raised more and more as tolerances are getting smaller and smaller.  So what do you do when presented with a simple flatness tolerance and a surface to measure?

Let’s start with the definition.  In the US, we have the standard ASME Y14.5 “Dimensioning and Tolerancing.”  At the time of this blog post, the 2009 edition is active.  In Y14.5-2009 we see:

5.4.2 Flatness

Flatness is the condition of a surface or derived median plane having all elements in one plane.


But if we are going to measure flatness, we need to understand this definition a bit more. Specifically, “what’s an element?”  Unfortunately, the work “element” is a generic term used throughout Y14.5-2009 to imply anything of interest.  For example, “elements” could be points, circular sections, linear sections, etc. 

In the case of flatness, it makes the most sense from the measurement perspective to consider elements as the data points representing the surface.  Thus, the metrology interpretation of 5.4.2 would be something like “flatness is the condition of a surface whereby all of the points are in one plane”.


But here’s the big question… ok there are actually two (related) questions?

How many points does it take?   (Sampling)


What kind of filtering/processing do we do with the points?  (Smoothing)


The fundamental challenge with understanding “flatness” boils down to “what do you really mean by flatness?”  For example, this surface can have two, very different flatness values depending on sampling and smoothing:

The reported flatness value depends on the surface wavelengths of interest.  For the above surface, the unfiltered flatness (left) is many times greater than the smoothed flatness (right).  The data on the left side includes all wavelengths.  The data on the right side is smoothed to include only the longer wavelengths.


Does Y14.5-2009 tell us anything about data points and smoothing?

Yes and no. 

Rule #1 (section 2.7.1) famously states that perfect form is required at the maximum material condition.  From the measurement perspective this implies the control of “all unfiltered data points”.  Rule #1 is also referenced in the Y14.5 section 5.2 regarding “form control”.     

At the time of this writing, an upcoming revision of ASME Y14.5 is planned to include further clarification.  This revision will contain a note indicating that small features such “surface texture” and “flaws” should be included in analysis of things like flatness.


CONCLUSION from Y14.5-2009:  Infinite number of data points.  No smoothing.

Practically speaking, we cannot measure all the way down to the last atom.  Therefore, let’s limit our measurement of flatness to go down to the shortest limit of surface roughness.  A typical roughness measurement is based on using a diamond stylus tip with a 2 micrometer radius (or smaller) and with a data point spacing of no more than 0.5 micrometers.  This can result in an overwhelming number of data points as we consider flatness.  A pencil tip of approximately 0.5mm by 0.5 mm puts this in perspective:


But there’s more to the story…

There is a bit of ambiguity in the US standards as we look a bit deeper.  For example, ASME Y14.5 refers to ASME Y14.5.1-1994 for “Mathematical Definition of Dimensioning and Tolerancing Principles.”   ASME Y14.5.1 has a section (2.1.1) on “Establishing the Surface Points.”  This section indicates that there is certain amount of filtering needed to separate dimensional features from “micro” features like roughness.  The context implies that “dimensional features” are those features defined in Y14.5… which includes flatness.


CONCLUSION from Y14.5.1-1994: Smooth out the roughness.


But wait!  ASME Y14.5.1-1994 also refers to ASME B46.1.  What does B46.1 say on this topic?

Along with the above mentioned note (2.1.1) ASME Y14.5.1 also gives reference to the use of “smoothing functions defined in ASME B46.1”

In the 1990’s ASME B46.1 (v1995 Section 1.2.2) included the concept of “error of Form” as being a type of shape.  It described “errors of form” as being comprised of “widely spaced deviations which are not included in surface texture”.   Since “surface texture” includes roughness and waviness, it implies that “form” is made up of wavelengths longer than waviness.  This results in a “3 domain” definition:


CONCLUSION from ASME B46.1-1995: Smooth out the roughness AND waviness for flatness.


So what is “flatness” in the United States?

In the US we have 3 candidates for flatness per ASME standards:  ASME Y14.5, ASME Y14.5.1, ASME B46.1 (pictured in order left to right)


What do International Standards say about flatness?

In its early drafts, ISO 12781-2 proposed the use of a 0.8 mm cutoff wavelength to remove short wavelengths (i.e. roughness) from flatness.  This 0.8 mm value was chosen since it is the most common surface roughness cutoff wavelength.  Thus, the idea was that there would be two domains:  the roughness domain and the “form” domain.

This approach, whereby roughness is removed from flatness, is much more manageable than trying to use an infinite number of points with no filtering.  However, it can still be difficult to achieve on very large surfaces.  To transmit wavelengths longer than 0.8 mm, a 0.5 mm tip radius is required and the data point spacing must be no greater than 0.15 mm.   Based on the implications of using this on large surfaces, this proposal was rejected by International balloting and the 0.8 mm filter cutoff was removed.  A statement was added to the Standard’s introduction indicating:

At the current state of development, ISO TC 213 has not been able to reach a consensus on defaults for filter UPR, probe tip radius and method of association (reference plane). This means that a flatness specification must explicitly state which values are to be used for these specification operations in order for it to be unique.


CONCLUSION from ISO 12781-2 Early drafts: Smooth out the roughness.

It is important to note that all of the ISO discussions involved the smoothing out of roughness.  The dissention regarding the 0.8 mm filter came from those that were dealing with large surfaces and they actually required more smoothing to extract their shapes of interest.  This ultimately led to the note in the introduction to 12781-2 (2011).

CONCLUSION from ISO 12781-2 (2011): You must specify filtering, sampling, etc.


So… what should you do about this?

First off you need to determine what you mean by flatness.   Are you interested in the general trends of the surface as flatness with aspects being controlled by waviness and/or roughness?  Do you need all unfiltered points captured? 

Ultimately the ISO 12781-2 (2011) standard gives us the best possible direction.  In section 4.1 it tells us that the proper (unambiguous) specification of flatness “defines the transmission band”.  Thus, if you want to control flatness – you need to indicate a wavelength band that describes what you mean by “flatness”.  When presented with a flatness tolerance and a surface to measure, you must ask for more information.  The tolerance limit alone is ambigous.


Still unsure?

For more information and for help specifying and measuring flatness and other surface-related challenges, contact Digital Metrology today!


Harmonic Analysis of Roundness Data

What is Harmonic Analysis?

“Harmonic Analysis” sounds impressive doesn’t it?  Don’t let it scare you – it’s a fancy mathematical way of describing things that move (or shake) back and forth. If a pen was oval it would bounce twice per revolution if you rolled it on your desktop.  That’s a 2nd order harmonic.  (2nd order means 2x per revolution).  Many wood pencils are 6 sided.  Thus, they have a strong waveform with bumps occurring six times per revolution.  We can also say this a different way:  the pencil has a strong amplitude at six “undulations per revolution” (UPR).  An “undulation” can be thought of as a sine wave or a “cycle”.

 With those sentences, we just did “harmonic analysis” … using words.  We can also do this with math and software.

 Harmonic analysis looks at your measured data and tells you “how many times” the bumps occur in each revolution.  In fact, harmonic analysis actually looks at every frequency (every UPR) and reports how tall the bumps are for each of them.  That’s a powerful analysis.


Digging in a litter deeper…

This ugly shape in the center of this figure is made up of 4 different “pure shapes” that are added together.



Each of the four shapes around the edges represent a “harmonic”.  A harmonic is sine wave that occurs a certain number of times per revolution.  This center shape is made up of four different harmonics.  In this example, the harmonics are 2nd order (lower left), 3rd order (upper left), 17th order (upper right) and 31st order (lower right).

Harmonic analysis works in the opposite direction.  Instead of combining simple shapes to make a complicated shape – harmonic analysis starts with the complicate shape and breaks it into simple sine waves.  Typically, a graph or table is provided to tell you exactly how much of each harmonic is present in the data.  In this example we see a bar graph.  The taller bars indicate that there is “more” of that specific harmonic.  For our current example with 4 harmonics, the bar graph would look like this:

The 4 bars indicate that there is significant “amplitude” (or “height”) at these frequencies.   On thing that this graph often leaves out is the “phase” of the harmonic.  For example, the 2nd order (oval) component is currently oriented with the first peak at 45°.  The phase of each harmonic tells us where the first peak occurs.

One thing that needs to be managed with harmonic analysis is what you mean by the word “amplitude”.  Some people specify “mean-to-peak” amplitude while others use “peak-to-valley” amplitude.  It is important to get this term right – especially if you are setting a tolerance on peak-to-valley and your supplier is measuring mean-to-peak!


What can we do with this?

One thing that we can do with harmonics is to put tolerances on them.  This can be done with a table of harmonic limits.  Or in other cases an equation that sets a limit “curve”.  Below is an example from OmniRound with the harmonic limit curve being set by an equation.


OmniRound highlights the bars that exceed the tolerance limit.  Furthermore, OmniRound detects the worst harmonic (the one that uses the most of its tolerance) and reports the amount of tolerance it consumes.  This is shown has the Harmonic Consumption (Harm.Cons.) parameter.  In the above example, the chatter limit is exceeded at 41 UPR.  This is shown on the graph and in the “Harm.Cons. UPR” parameter. 

The Harm.Cons. parameter is a powerful way to combine harmonic analysis across hundreds of UPR and turn it into a single number.  For a surface to be acceptable, all harmonics must lie below the tolerance curve.  Thus, the Harm.Cons parameter must be below 100%.



Harmonic analysis is a powerful tool for understanding your roundness data.  Check out Digital Metrology’s “OmniRound” software and get started with your own harmonic analysis for your data sets!

Tip Radius Selection for Roundness

The measurement of roundness is often debated.  

Many of these debates are centered around the tip radius and filtering used in the measurement.  This debate is further complicated by the fact that the typical roundness graph distorts reality.

Here is the same roundness data plotted at varying magnifications…

When we look at a roundness graph for an outside diameter measurement we see locally concave areas.  Often these areas look rather sharp.  This is due to the polar plot scaling and the magnification of “errors” rather than the magnification of the “size” .

The rightmost plot appears as though there are large concave areas on this component.  However, this data is from a precision gage ball.  The actual workpiece has no concave areas as the polar plot seems to indicate.  In fact the actual surface is completely convex and, given the magnitude of the roundness errors, a flat follower would have no problem measuring the roundness of this surface.

One other problem arises from a polar plot and it’s “vanishing point” at the center of the plot.  Since negative values all approach the center of the plot, they tend to “compress”.  Positive values tend to “fan out”.  This gives the perception that inward features are much sharper than outward features:


When we increase the scale, we see a large (apparent) difference between peak and valley curvatures:


On the right most graph (above) we get the perception that the valley bottom becomes a sharp point.  However, this is not the case with the actual data.  

So let’s look at sine waves and think about the concept of “radius of curvature”…


Linear Sine Waves and their Radii

The radius of curvature of a sine wave (either peak or valley) is given by:



RSINE        the smallest radius of the sine wave (for example at the valley).  See Figure A4.

l             the wavelength of the sine wave

A             the amplitude of the sine wave


Since we are dealing with circular parts, the wavelength, l, is based on:


RPART       the radius of the part being measured

fUPR         the frequency of interest expressed in undulations per revolution (UPR)

which ultimately gives the sine wave radius on a circular part as:


What about Inside and Outside Diameters?

I’m glad you asked.  

If we are measuring an outside diameter, we can use a larger tip radius to see all of the features of a given sine wave.  If we are measuring an inside diameter we will need a smaller radius in order to see the same features:  This figure shows a sine wave wrapped around a circle.  In doing so, the spacing between the peaks of the sine wave are effectively “widened” as compared to a sine wave on a straight reference line:

Ideally, we would like to combine the radius of curvature of the part with the radius of curvature of the sine wave.  Unfortunately, radius values do not mathematically combine, but curvature values do combine.  Mathematically, curvature is the inverse of the radius of curvature:


For an outside diameter, the largest allowable tip has a curvature of:


For an inside diameter, the curvatures add:

Thus for an Outside Diameter:

Outside diameter considerations:


Similarly, for an Inside Diameter:


A             the amplitude of the sine wave

RPART       the radius of the part being measured

fUPR         the frequency of interest expressed in undulations per revolution (UPR)


But what about the filter?


The above defined tip radius will actually touch the bottom of the sine wave, however it will only occur at a discrete point and will cause a “cusp” or “sharp corner”.  


Since the cusp is very sharp, it will be filtered and some of the depth will be lost.  Thus, a smaller tip should be used in order to gain additional confidence.  A reduction of 25% may be adequate as a safety margin.


How about an example?

A 10 mm inside diameter has a roundness limit of 0.005 mm with a 50 UPR filter cutoff:

Using the inside diameter equation:

… and assuming that all of the peak-to-valley roundness error occurs at the highest frequency we have:


Inserting values:

Which gives a max tip radius of:


A 25% safety factor (multiply by 0.75) gives:



Roundness may continue to be debated.  But this discussion shows that the relationship between surface waveforms and the tip radius can be understood.  

For more help understanding your toughest metrology challenges (and help with settling your debates) contact Digital Metrology today!