About paper, about mobile phone screens and fingertips

Up close it's easy to see why paper is good at holding on to the grainy bits of charcoal or graphite as they break off. 

Rag paper

Cartridge paper

The fact is that a micro-world is going on all the time whilst we are drawing but is usually just below the levels of our perceptual awareness. Whatever we are doing, our consciousness of what is happening is very limited and we are therefore unaware of many of the consequences of our actions.

However we don't use pencils as much as we use our mobile phones in present day society, so I have decided to look at the contact made between the human fingertip and a mobile phone touchscreen. But before I do that I think its important to remind everyone that in many ways using our fingers to make drawings on our mobile phone screens links us back to a time of drawing with our fingers in mud or sand, the brush and the pencil being less sensitive devices that we are learning to do without. 

A close up view of a finger and a mobile screen

Touch relies on some pretty sophisticated physiology. Using a mobile phone requires using your fingertips to do lots of swiping and sliding (haptic tasks), therefore an awareness of how touch works is important if you are to begin thinking about what is happening just beneath the everyday surface of your consciousness. Touching something else is central to the process of change, it is contact that forms other things; for instance as wind blown dust touches a rock it slowly and inexorably begins the process of erosion, or if you want to get technical, aeolian processes are at work.

This sandstone outcrop has been carved by the wind

Contact can be rough and sudden, a bullet cuts into a wall as its force is spent, or slow and gentle, as the tiny shards of shells of micro organisms drift down through sea waters and settle to build up what will one day become chalk deposits or spasmodic, like the breaking off of flakes of charcoal as an artist makes a drawing on sheets of paper. 

Think of those things that you or others have touched over and over again, the polished toe of a venerated statue, the handle of a door that has been used for many years, you don't notice a change at the time, but something always happens when one thing comes into contact with another. 

The polished toes of a statue of bishop Grgur 

Touch shapes things, both the thing being touched and the touching finger is shaped by the contact. But we hardly notice what happens. It takes thousands of touches to polish Grgur's toes and each of those touches would have knocked a few cells off the end of someone's fingertip. Each stroke of your fingers across the screen of your mobile is also making things happen, a very complex series of changes are taking place, and you are being changed by the contact, so lets see how.

Human fingertips can feel the difference between a smooth surface and one with a pattern embedded just 13 nanometres deep, or about a human hair width. Epidermal ridges on the surface of our fingertips allow us to differentiate between a wide range of textures, materials, temperatures, and pressures. We all have a unique pattern of these fingerprints but the pattern is not crucial to the function. Their importance is that just underneath the ridges are mechanoreceptors that respond to tactile stimulus. Friction caused by movement of the fingertip along the surface of the computer screen stimulates these mechanoreceptors, which then transmit tactile information to the brain. Your skin has three layers and receptors that let the body sense touch are located in the top two layers of the skin; the epidermis and the dermis.

The epidermis, the outermost layer of skin, also provides a waterproof barrier and creates our skin tone. The dermis, beneath the epidermis, contains tough connective tissue, usually hair follicles (our fingertips are hairless) and sweat glands and the deeper subcutaneous tissue (hypodermis) is made of fat and connective tissue. 

The epidermis has itself five different layers. Stratum basale, stratum spinosum, stratum granulosum, stratum lucidum and stratum corneum and if you look at the diagram below you can see how the shape of the cells allows the top layers to flake off, (dead cells) providing an ever changing and very interesting surface that is central to what happens when we touch anything. We in effect leave traces of ourselves on whatever surfaces we come into contact with. Just as the charcoal is flaked off on contact with the drawing paper, we are ourselves being flaked off as we drag our fingers across the surface of our mobile phone screens. This is an on-going process, new cells being formed at the junction between the dermis and epidermis, which slowly work their way towards the surface of the skin ready to be released in a process that constantly replaces shed skin cells. 

The epidermis

The receptors in our fingers are all part of the body’s somatosensory system, a huge network of nerve endings and touch receptors. This system is responsible for all the sensations we feel; cold, hot, smooth, rough, pressure, tickle, itch, pain, vibrations, and more. The four main types of receptors are; mechanoreceptors, thermoreceptors, pain receptors, and proprioceptors.

Different receptors collect different information, for instance a rapidly adapting receptor can respond to a change in stimulus very quickly, which means that it can sense right away when the skin is touching an object and when it stops touching that object. However, these receptors can’t sense how long the skin is touching an object. Slowly adapting sensors do not respond to a change in stimulus very quickly. These are very good at sensing the continuous pressure of an object touching or indenting the skin but are not very good at sensing when the stimulus started or ended. Both rapid and slow receptors respond to changes in pressure as you push your fingers across the surface of your mobile phone, these are ‘mechanoreceptors’; receptors that respond to sensations such as pressure, vibrations and texture and their only function is to perceive indentions and vibrations as they effect the skin. The four types are; Merkel’s disks, Meissner’s corpuscles, Ruffini’s corpuscles, and Pacinian corpuscles. The most sensitive being Merkel’s disks and Meissner’s corpuscles. Merkel’s disks being slowly adapting receptors and Meissner’s corpuscles rapidly adapting receptors, enabling your skin to perceive both when you begin touching something and how long the object is touching the skin. Therefore that moment when you first touch your mobile phone screen is signalled to the brain by your Meissner corpuscles, Merkel’s disks then take over telling the brain that this is a continuous activity and when you take your finger off the screen, Meissner corpuscles signal the change. This is all going on in the epidermis and outer layers of the dermis and located deeper in the dermis and along joints, tendons, and muscles of your finger are Ruffini’s corpuscles and Pacinian corpuscles. These mechanoreceptors can feel sensations such as vibrations traveling down bones and tendons, rotational movement of limbs, and the stretching of skin. These are helping you control that finger and apply changes in direction and pressure to it as you decide to do the things you need to do with your phone. However that is not all that’s happening, the screen might be warm or cool to your touch, depending on perhaps how the system is working, for instance a flaw could be causing the device to overheat and you will be made aware of the danger by ‘thermoreceptors’; receptors that pick up sensations related to the temperature of objects the skin feels. They are found in the dermis layer of the skin and they are divided into hot and cold receptors. 

Your phone’s screen may however be broken, and if so you might prick your finger on one of the edges of the broken screen and in this case ‘pain receptors’ will come into play; these are the nocireceptors. They can detect pain that is caused by mechanical stimuli (cut or scrape), thermal stimuli (burn), or chemical stimuli (poison from an insect sting).These receptors cause a feeling of sharp pain to encourage you to quickly move away from a harmful stimulus such as a broken piece of glass or a hot stove top. They also have receptors that cause a dull pain in an area that has been injured to encourage you not to use or touch that limb or body part until the damaged area has healed. 

There are other receptors that are vital to your holding a phone and these are the ‘proprioceptors’ or one’s own awareness-of-self receptors. They sense the position of the different parts of the body in relation to each other and the surrounding environment. Proprioceptors are found in tendons, muscles, and joints. These cells detect changes in muscle length and muscle tension, i.e. without them you would be dropping your phone all the time because you would become totally uncoordinated. 

While many receptors have specific functions to help us perceive different touch sensations, you never find just one type active at any one time. When using your phone, your hand can perceive many different sensations just by holding it.

Mechanoreceptors can sense that your hand is stretching around your mobile, whilst at the same time sensing that pressure is being exerted to both hold the phone and push your fingers around the phone's screen surface. 

It’s also important to remember that when your fingertip touches that screen, the mechanoreceptors that are activated begin a chain of events by signaling to the nearest neuron that they touched something. This neuron then transmits this message to the next neuron which gets passed on to the next one and on it goes until the message gets to the brain. Now the brain can process information received about the surface that your hand touched and send messages back to your hand via this same pathway to let the hand know what to do next in order for the brain to carry on getting more information. 

Your brain though gets wildly disproportionate information about touch from different parts of your body. The fingers you are using are packed with sensors, but other parts of your body, such as your back, have very few, reflecting the fact that we have very different relationships with various parts of our body. 

How touch sensors are represented in the brain

The part of your brain that processes touch information has embedded within it a very distorted map of your body. It over-represents areas that have lots of fine touch receptors (like the face, the lips, the tongue, and the fingers) and under-represents areas that don't have many receptors. This ‘map’ is constantly changing, because areas of the touch-sensing parts of your brain that you use a lot tend to expand and take over neighbouring territory. Therefore the area of your brain that processes information from fingers used to control your mobile, will expand the more you use them. 

This relationship that we have with the mobile is very important because there is another system at work, the emotional touch system, which is mediated by special sensors called C tactile fibers, and it conveys information much more slowly. It's vague in terms of where the touch is happening, but it sends information to a part of the brain called the posterior insula that is crucial for the development of socially-bonding touches; such as a hug, holding hands, or sexual foreplay. 

This is where it gets creepy; the touch screen of your mobile is looked at with the same intensity as you would look at a person that you were sexually attracted to, when you brush your fingers over the touch sensitive screen, it is exactly that, touch sensitive, just like those people who we have intimate relations with. The screen is made in layers just like human skin, it’s smooth texture is very like skin, it gives slightly as you touch it, just like skin, its slightly warm because you keep it on your person and most importantly you have built up an emotional relationship with it just as you would another human being. Therefore unlike most other objects, engaging with your mobile can cause C tactile fibres to be triggered. You adjust the speed of your finger as you stroke, so your vibrational senses are able to detect slippage and friction, but you are not just seeking surface change, you are adding in an emotional feeling as you would when relating to another human being. This sort of makes sense because the mobile is mainly used to communicate with other humans, but because we invest so much of our emotional energy in this device, we have little left for other humans. 

But what about the thing being touched? The mobile phone screen often gets pretty battered in the contact improvisation dance it has with human beings. 

 

A detail of Duchamp's large glass

Formally the broken glass is reminiscent of Duchamp's 'Bride Stripped Bare' which was also broken by accident, and the results finally embraced as a chance action that seemed to be an acceptable part of the work's journey. An example of contact improvisation, where  the artist accepted that the elements outside of his control were giving as much to the work as himself. 

However, the screen is not designed to be broken, it is designed to be touched and there are various ways that designers have come up with to make sure that when you touch a screen the information is transmitted to a phone's operating system. 

The touch screen on a mobile phone is a display that can locate the presence and location of a touch within the display area. It needs a minimum of three components to be able to get information to the phone's operating system, a touch sensor, controller and software driver. 

I was still at art college when the first touch sensor was developed in 1971 by Sam Hurst at the university of Kentucky, so you are working with technology that is already 50 years old. 

The touch sensor is usually a clear glass panel with a touch responsive area, this is placed over a display screen, so that the responsive area covers the viewable area of the screen.  I.e. if you can see an icon you can touch the touch responsive area above it.  An electric current is made to pass through the touch sensor, this carries signals which are changed when pressure is exerted on the screen and the change in a signal is used to determine the location of touch on the screen. The controller connects the touch sensor to the computer operating system in your phone. It takes data from the touch sensor and translates it into information that the computer can understand. 

A controller component, in this case a MPR121, is designed to work with the specific technology that the screen uses to collect data and has lots of pin connections because of the need to connect up a complicated array of sensors all collecting information about where your finger is touching the screen. In this case the technology is capacitive, which is the system an I-Phone uses. 

Diagrams showing typical connections and links to electrical power supply 

Before I move on to look at the workings in more detail, it's I hope worth reminding you how important drawing is to coming to an understanding of all this complex information, the diagrams above are both beautiful and informative, and cross over disciplines. 

The software driver allows the touch screen and the computer to work together. This is very like what happens when you add a new printer to your PC, you always need to install new driver software in order to get access to the printer. The driver ensures that a touch on the mobile screen operates the same way that clicking and moving your mouse operates in relation to a computer monitor.  It tells the mobile phone’s operating system how to interpret the touch information event that is sent from the controller. 

A capacitive screen such as that used on an I-Phone, consists of an insulator like glass, coated with a transparent conductor like ITO, (Indium-tin-oxide). Touch then distorts the screen's electromagnetic field, which is measured as changes in signal intensity along both x and y axis. More than one layer is used so that separate information can be collected from each axis and then coordinate points plotted. 

Projected capacitive panels have multiple sensors, which means that they can detect more than one pressure difference at the same time, i.e. you can use more than one finger at the same time. 

When a fingertip comes into contact with a capacitative touchscreen, it uses the electrostatic conductivity of the human body as a means for input. Unlike resistive type touchscreens, electrostatic capacitive touchscreens are highly responsive, but if you turn your finger over and try and use your fingernail, you will find that nothing happens. This is a good way to test out what sort of screen technology your phone uses, it also explains why I-Phones are so sensitive to wet conditions. 

So if you are making a drawing on your I-Phone, your own body's electricity is being used at the point of contact. Capacitive touch-screen technology means that you aren't limited to simply pressing the screen in one place. The iPhone can detect the difference between your pressing the screen with one, two, three or four fingers. It can also detect gestures such as swiping or pinching. This sensitivity gives you a much wider range of controls for each individual application. It also helps make the user interface much more intuitive. For example, programmers can map your finger swipes to scrolling through a long page, or pulling two fingers apart to zoom in on an image. However for drawing purposes when you might want more pressure sensitivity a resistive type touchscreen could be better. Going back to that graphite pencil point breaking off onto the surface of the paper, its H or B grade could be seen as analogous to whether the touch screen is capacitive or resistive. 

Just as your internal sensing system uses a variety of inputs to determine what is happening at your fingertip, the mobile phone is using a variety of inputs to determine what is going on at the point of contact, so when you draw on your phone, as with all media, it is partly you and partly the media that shapes what happens.

Technical drawing of an I Phone

However the I-Phone was once an idea in someone's mind, and as such this was realised as a technical drawing before it was manufactured. Therefore you might want to think about its fascinating 'thing' history in more detail, especially as information begins to become entangled into a knot of correspondences. For instance indium-tin-oxide (ITO) a material that is used in mobile phones because of its electrical conductivity and optical transparency  as well as the ease with which it can be deposited as a thin film, is also very expensive. The high cost and limited supply of indium is a real problem, as well as the fact that during the process of mining, production and reclamation, workers are exposed to it. It is mainly mined in China, Japan, USA, the Republic of Korea, and Canada. Indium lung disease is developed through contact with indium containing dusts and there are several proven cases of workers coming into contact with indium, developing conditions such as pulmonary fibrosis, emphysema, and granulomas. As I drag my fingertip across my phone's surface in order to make a small drawing in its note application, at the same time a mine worker is developing granulomas when his or her immune system attempts to wall off substances it perceives as foreign but is unable to eliminate because their body doesn't usually have to ward off long term exposure to indium. 

A granuloma

As usual with these long posts I'm beginning to ramble, but hopefully the point has been made. As I pointed out at the beginning of this post, our consciousness of what is happening is very limited and we are unaware of most of the consequences of our actions. Drawing on a mobile phone is no different to drawing with a pencil, both are about the contact made between one thing and another, both are media specific, both on contact with something else cause change to happen at a macro and a micro level and our awareness of all these things is always partial. Pencils have long chains of conditionality* behind them and so do mobile phones and as people that use both of them, the more aware you are about what goes on the more choices there are as to how and why you might use them. 

* In Buddhist thought Pratityasamutpada means 'dependent origination' or 'conditionality'.  It can best be understood as the interconnectedness of all existence or each and every action has a consequence. 

See also:

The macro and the micro
The pencil and sustainability
More thoughts about paper
Pencils and erasers
Drawing hands

Swell paper for making touch drawings