Many publications, both in print and on the web, will test a product and give you, the reader, a definitive rundown on how good the product is from that single test. Often the results are at a variance to one another and sometimes they can be totally contradictory.
So are these differences in the reviewer’s objectiveness and methods or are there true differences in the products? Is quality control at fault or is it bias amongst the journalists?
In this rarely carried out test I took a detailed look at three lenses that should, to all intents and purposes, be identical.
Why a rarely carried out test? When a lens is announced, everybody wants a look at it. Manufacturers, or their distributors, only have a set number of ‘samples’ to go around and there is always a scramble amongst reviewers to get 
one first. A couple of months down the line the next announcement has been made and the previous ‘latest’ has been forgotten. Except, of course, by you, the reader. You have read the reviews, saved your hard earned, and waited patiently for your local stockist to actually get that model of lens in.
Most manufacturers would shudder at the request I made for three lenses, all of the same model, picked out at random. In fact, all the ones I asked did!
Sigma though, thought about it for a minute and agreed to allow me into their warehouse and select three examples of the same lens at random off the shelf. In other words, the lenses were taken out of the supply chain that ends with you, the customer.
The lens I chose was Sigma’s popular 105mm f/2.8 EX DG Macro. Although this lens has been around for some while, it has had a recent upgrade to DG classification, making it, in effect, a ‘new’ lens. From their stock I took one from the front, one from the back and one from well down the stack. The serial numbers, once I had them back here to unpack, differed widely enough to ascertain that we had a random sample. The boxes were unopened, just as they had arrived from the factory in Japan. The lenses were all the same Canon mount and the tests were all carried out on the same EOS 20D camera. The test target I employed is the ISO12233 compliant slant edge test target and the resultant files were run through a program called Imatest® after being shot in RAW format and converted with no sharpening. Read on to find out the results.
- Product: Sigma EX 105mm f/2.8 DG Macro
- Elements/Construction: 11 elements/10 groups
- Angle of view: 23.3° (35mm full frame)
- Max aperture: f/2.8
- Min aperture: f/45 (Canon, Sigma, KM. F/32 Nikon, Pentax mounts)
- Min focusing distance: 31.2cm (12.2 in)
- Corresponding mounts: (AF) Sigma, Canon, Nikon, Pentax, KM/Sony
- Filter size: 58mm
- Length: 95mm (3.7 in)
- Diameter: 74mm (2.9 in)
- Weight : 450gms (15.8 oz)
Along with the lens is a dedicated screw-in type of lens hood. This item alone is totally different to earlier models that had an all-metal bucket type hood. Lens cap and mount cap are also supplied.
The lens has a solid feel to it and the matt black finish looks both durable and modern. On closer examination, the broad manual focus ring has a backward and forward movement of around 5mm with a positive click stop at each end of the travel. This movement disengages the ring from the autofocus mechanism and allows it to autofocus without having to drag the ring around with it. Focussing extends the front element on a single trombone system that does not rotate during the process. This is handy when using filters, especially polarising or graduated ones that need to stay correctly orientated. The focussing system extends this trombone by as much as 51mm when going down to the 1:1 distance of 31.2cm but it is nicely engineered and has virtually no play in it, even at the maximum extension.
The centre of the lens boasts a neat little distance widow, marked in both metres and feet along with a token depth of field scale marked at f/32. Much more useful are the markings on the barrel extension, giving a good representation of the reproduction ratio being obtained. These correspond with the M/AF markings on the main barrel, which are in gold for manual focus and white for autofocus, the change occurring when the focus ring is shifted.
For an f/2.8 lens, the front element is surprisingly small with a filter fitting of just 58mm and it is this thread that takes the newly designed hood, now made from engineering plastic and a much better fit, especially when reversed for transport/packing. It does, however, retain metal threads, both when fitting onto the lens as well as at the outer end where a second thread, this time in the popular 77mm size, can take further filters. This is handy if you already have these accessories for a wide-angle zoom or the like.
Generally, and this applies to all three examples, the handling felt smooth and positive with switches having that clean movement that inspires confidence in how long they will last. Handling the three lenses, there was no way they could be told apart physically unless you checked the serial number stamped on the underside of the barrel.
The results so far are not particularly surprising, as I would not have expected to be able to distinguish a difference simply by looking and handling the lenses. Where I really expected differences is in the optical performances of the lenses. This is far more likely as the manufacture, grinding and polishing of the glass elements, along with the critical positioning of those elements within the barrel of the lens is far more prone to errors than any other process the objective goes through. A hair or speck of dirt in the wrong place can tip an element enough to cause a problem and everything has to go together in a dust free environment.
Having tested lenses of all types and under various conditions, I knew that there would be differences in the performances of the three examples. Because there are so many variables, even in test conditions where temperature and humidity can vary from day to day, none of my previous data was really relevant.
So the tests were carried out at the same time without moving the camera, a Canon EOS 20D mounted on a Manfrotto 055PROB tripod with a 486MGRC4 head and changing the lenses for both the centre and edge target shots. An image of the slant edge target was taken at each full F-stop for both the centre and edge of the frame, making 19 images per lens when you include the single shot of a separate target needed for the distortion test.
Focussing was set with the autofocus turned on for the first shot and then disabled for the remaining shots in that sequence of apertures in order to avoid variations in refocusing. Files were shot in the RAW format and converted to .tif format in Pixmantec’s Rawshooter Essentials. No sharpening was applied.
Next up is the resolution test and in all honesty, this is where my eyebrows started to rise. Not because of the variations, but because of the consistency of the results especially from f/5.6 onwards. Even wide open, where the accuracy of the focussing can have an effect on the results, the results were close enough for me not to have any undue concerns with the first of the three having the softest wide aperture performance, but even this one was good enough for most purposes.
An anomaly I have seen before on lenses that are designed to be used on full frame 35mm film and sensors is that when tested on cropped sensors such as that on the EOS 20D, the ‘edges’ occasionally outperform the centre of the optic. This does not mean something is wrong with the centre, but rather, it is a good sign that the lens will retain good resolution marks right out to the edges of the full, 35mm sized frame. From f/5.6 onwards, all three of the lenses showed this characteristic very consistently. At f/8, the figures achieved by all three of the units were virtually indistinguishable and then only within the fine tolerances of the test programme. Nobody would be able to tell the difference by eye.
The digital coatings that have been developed to reduce incidences of ghosting and flare that are accentuated by digital sensors reflecting light back into the lens barrel, where it can bounce about causing havoc, have had a beneficial side effect in Sigma’s case, improving the contrast of the lenses they have been applied to. Good contrast improves the apparent sharpness of a lens, or more correctly, the resultant image.
Flare can, of course, always be induced in extreme conditions, but even shooting with the sun in the corner of the frame (with my eyes protected to save harming them) the flare was restricted just to the small area around it and did not bounce around inside the lens affecting the whole frame. Although not possible to easily measure, the control was similar for all three lenses. The flare assessment was carried out without the supplied hood fitted and with the hood having the same matt black finish on the inside as the whole of the lens on the outside, this will further cut down any occurrences.
Chromatic aberrations, which I can and do measure, were recorded at insignificant levels, again by all three lenses. By insignificant, I mean that they are below 0.5 of a pixel down to f/11 and below 1 pixel from f/16 onwards, where reciprocation starts to become the limiting factor. CA needs to exceed 1 pixel in order to become apparent to the eye and measurements of up to 1.5 pixels are normally deemed acceptable.
Macro lenses, by their very nature, tend to be used well stopped down in order to maximise the depth of field. The slight variation in the wide-open performance between the three examples is not enough to be a concern, especially as the average figures are very close to one another. There is one lens, number 3 of the examples, which appears to have a slight edge at the wide-open apertures, but in practical use the difference would be very hard to detect by eye. From f/5.6 onwards, there are no real differences between the three optics in any of the areas that we tested, a truly remarkable performance!
Verdict
To say I was surprised by the consistency of the results is somewhat of an understatement. Having conducted tests in the past, albeit hardly ever on a direct head to head basis except on second hand lenses that varied enormously, I expected a much greater variation.
Another consideration I have not yet noted is the lens tested is a prime lens, the construction of which has fewer variables than a more sophisticated zoom. However, the results still show an extremely high standard of quality control in modern manufacturing methods, and for that, Sigma at least, must be commended.
I started off this piece by trying to exonerate journalists from the bias that they are sometimes accused of, hoping to lay the blame squarely at the manufacturers feet. I seem to have failed miserably! The results here show that it is clearly not the case and the equipment that you, the consumer, gets off the shelf of your local supplier is subject to a rigorous system of quality control.
Perhaps it is the other way round and the manufacturers rush to get early samples out to members of the press corps means that the consistency has not yet been achieved. Or is that trying to move the blame again?
Whichever, I feel that the exercise has proved well worthwhile and will have helped your piece of mind when purchasing a new item of equipment, certainly from Sigma.
The basis of this article first appeared in Issue 45 of Digital Photographer, a UK magazine.