CRITIQUE of the Department of Commerce report entitled "America's New Deficit, The Shortage of Information Technology Workers"


Billy E. Reed, Pres.

Contact e-mail (Robert Rivers, Chair)

Prepared by: Norman Matloff Department of Computer Science University of California at Davis

Nov. 8, 1997

On September 29, 1997, the Department of Commerce released a report, "America's New Deficit: The Shortage of Information Technology Workers," which claims a software labor shortage. The DOC report was basically requested by the Information Technology Association of America (ITAA), an industry trade group, and in fact DOC's writeup was for the most part a warmed-over version of ITAA's earlier report.

Below I list my reactions to the report. I consider it to be highly irresponsible and inaccurate, and sad to say reminiscent of the National Science Foundation report in the late 1980s which forecast a severe shortage of Ph.D.'s in science, only to admit now that there is a glut of such people. The report is a "rush to judgement," published at the demand of an industry trade group with obvious heavy financial interests in having an ever-expanding glut of computer programmers.

Title page:

On the report title itself:

I object to the title, in that it takes for granted the existence of a shortage, based only on a misleading report by an industry trade group which has financial interests in having a glut of programmers.

Page 1:

On reliance on ITAA and other employer claims:

Again, the report relies only on employer claims of a shortage (and in the Coopers and Lybrand case, did not even say whether the claimed shortage was serious or tiny). This is a major error, in that employers have an enormous vested interest in the salary cost savings which would be incurred from an ever-increasing oversupply of programmers.

Salary is an absolutely key issue with IT employers, as pointed out by prominent software development expert Edward Yourdon: `` takes very little capital to get started [in the software business]; indeed, the largest cost is the labor involved, which is why the issue of salaries is so important.''

One way to see the degree to which reduction of salary costs is of prime concern to employers is that the employers have admitted that a major motivation for their exploring the possibility of shipping their software development overseas is the lower salaries involved. Sun Microsystems, the Silicon Valley giant, even bragged to the Los Angeles Times that it had been able to hire programmers in Russia "at bargain prices." Note that the list of regions DOC cites for offshore outsourcing on page 15 -- India, Russia, Eastern Europe, East Asia and South Africa -- are regions with salary levels well below those of the U.S.

The ITAA, whose lobbying efforts led to, and formed the main foundation for, the DOC report, has been extremely active in recent years in lobbying Congress to maintain and increase the yearly quota of work visas for foreign programmers. The Department of Labor and others have documented extensive abuse of this visa program. Such workers are on average paid lower salaries than comparable domestic programmers. (Note that the term "domestic programmers" includes not only natives but also earlier immigrants, who of course deserve just as much concern over issues such as job displacement as do natives.)

On the number of computer science degrees granted:

This is highly misleading. Though the report concedes that one can do IT work without a computer science degree, it does not tell the whole truth here. MOST PROGRAMMERS DO NOT HAVE COMPUTER SCIENCE DEGREES -- AND THEY NEVER HAVE. It has ALWAYS been the case that programmers have degrees in a wide variety of fields, such as engineering, math, the sciences, economics, business and so on. For example, an NSF survey indicates that out of approximately 550,000 programmers slightly less than 200,000 had any 4 year degree at all, much less a CS degree.

Page 2:

On the Stanford Computer Industry Project:

The DOC report quotes the Stanford Computer Industry Project (SCIP), as if SCIP were yet another independent voice in this process. But the fact is that SCIP is funded by the computer industry, and is allied closely with ITAA.

On the disclaimer that increased immigration is the solution:

This again comes from the ITAA report, and is of course a highly disingenuous assertion made by a body which has been at the forefront of lobbying for lax immigration law concerning foreign computer programmers. It is certainly not the case that the ITAA has abandoned this goal; on the contrary, ITAA has recently called on Congress to increase the yearly quota of work visas.

Indeed, the ITAA push which resulted in the DOC report appears to be a smokescreen hiding increased immigration as the central agenda. One can foresee the chain of events to follow: After getting the DOC imprimatur "confirming" a claimed shortage of programmers, the ITAA/DOC will then likely propose a package to solve the "problem." The package will on face value be multifaceted, involving technical retraining programs, increased funding at the K-12 and university levels, increased outreach to women and minorities and so on -- but the proposal will conclude by saying, "We believe these measures will in the long term solve the shortage problem, but in the interim we need an increase in the yearly quota of work visas for foreign programmers."

Note by the way how this approach will co-opt one of the most powerful groups in Washington, the education lobby, by including funds for them.

Pages 3-4:

On job titles:

First of all, DOC does not mention here that ITAA considers anyone with, say a computer terminal on his/her desk, to be an "IT worker." This is a ridiculous mixing together of, for instance, of engineers with university degrees and technicians who might not have even finished high school. Such a mixing is especially insidious in view of the fact that ITAA then discusses a decling in production of university computer science degrees, comparing the numbers of such degrees with the claimed number of unfilled positions for "IT workers." This is statistical sleight-of-hand.

(In my remarks here, I address myself almost exclusively to computer programmers, i.e. developers of software, because that is what is really the crux of the matter. This is true even from ITAA's point of view, for instance in the sense that most of the imported foreign workers are programmers.)

The DOC report only hints slightly what ought to be stressed heavily: Those who write software tend to have many different job titles -- without any corresponding difference in meaning. The personnel on THE SAME TYPE OF SOFTWARE PROJECT might have the title Programmer with one employer, System Analyst with another and Software Engineer with a third.

DOC and SCIP also misunderstand the role of software for embedded systems used in cars, cell phones and so on. DOC/SCIP point outthat many programmers working on such software come from backgrounds other than computer science, but fail to state (and fail to understand) that THE SAME IS TRUE FOR ANY TYPE OF SOFTWARE. Again, MOST PROGRAMMERS DO NOT HAVE COMPUTER SCIENCE DEGREES.

Similarly, though one could say that the job categories delineated on page 4 are based on a grain of truth, the fact is that they are mainly "differences without a distinction." Contrary to the claims made by ITAA and implicitly by DOC, these workers are INTERCHANGEABLE. The only exception is the Computer Engineer category -- anyone can do software with minimal training, but hardware design truly requires extensive background, though again such background is not limited to those with computer engineering degrees by any means. But even that is only a small exception, in that the vast majority of jobs in the IT industry are in software, not hardware. For instance, of H-1B work-visa application listings for a period in 1994 in Texas, there were 139 software positions, with titles like Software Engineer and Programmer, compared to only 30 for positions titled Electrical Engineer, and again, even many of those EE positions are likely to be in software development.

Pages 5-8:

On projected growth rates for the various categories:

Again, the differences in job titles are virtually meaningless, and thus DOC's various comparisons in growth rates is a meaningless exercise.

On the differential needs for programmers among various industry sector:

DOC makes it sound here like a programmer in one sector could not work in another. This is entirely false, and by the way there are many programmers laid off from the defense industry, and they would do well in other sectors. Note for example that TRW, now well known for its consumer credit database, converted to that sector after the Los Angeles aerospace industry declined.

Page 9:

On the millenium problem:

An October 4, 1997 article in The Economist reported that expenses for solving the "Year 2000 Problem" are turning out to be far less than what the IT industry had projected. This again shows that the industry is incapable of assessing its own needs. The industry's capability of self-assessment is even more unreliable when it has a financial incentive to bias its assessment, as in the case at hand here, with a labor glut's implication for cheap labor.

The fact is that IT managers and executives tend to be people who were promoted from the technical ranks. Having done well in programming or other technical work does not imply that these people have good management skills. Studies have shown, for instance, that there is as much as a 20-to-1 ratio in productivitybetween the best and worst programmers, yet of course there is nowhere near that range in salaries; indeed, the studies have found for example that programmers who are twice as productive earn on average only 10% more in salary. In other words, IT employers are not good at assessing employee talent either.

Page 11:

On salary increases:

A major problem is that employers are overspecifying job requirements. A programmer who does not have experience in a list of specific software technologies, say the new Java programming language, will have his/her application for the job automatically rejected.

This has a greatly distortionary effect on salary averages. When an employer insists that an applicant for a programming position have a certain laundry list of software skills, the employer is then driving up salaries in that very narrow segment of the labor market. DOC's mention of a 20% increase in hourly wages for consultants, for instance, is an illustration of this, since consultants are usually hired precisely because of their knowledge of a particular software technology. The same comments apply (though via somewhat different routes) for the other statements DOC makes on salaries here.

In other words, although salaries have risen in certain narrow segments of the software labor market, this is not the case in general. The fact is that figures compiled by the Bureau of Labor Statistics show no significant increase in wages from 1995-1996. Even DOC describes salaries for new graduates as having only "nudged" up.

The claim that there have been large across-the-board increases in programmer salaries would come as an unfunny joke to my wife and other programmers who work at corporation "X", a large and well-known Silicon Valley firm.

Moreover, Silicon Valley, where employers scream the loudest about a shortage, is famous for making its programmers put in large amounts of overtime. If there were a true shortage, programmers would be able to negotiate not just lucrative salaries but also favorable policies on overtime, which clearly is not happening.

This employer overemphasis on specific software skills is of absolutely central importance. THERE IS NO SHORTAGE OF COMPUTER PROGRAMMERS, but rather a "shortage" of programmers with specific skills. Microsoft, for instance, stated earlier this year that it hires only 2% of its applicants for software positions -- a rate which is absolutely inconsistent with the claim made by the industry (including Microsoft) that they are desperately short of people. If they were desperate to hire, they could not be this picky.

The salary increases in certain narrow segments of the market show that employers are shooting themselves in the foot by their policy of insisting that programmers who apply for software positions have experience in specific software technologies. And such a policy is unwarranted in terms of success of a software project. Any competent programmer can become productive in a new software technology in less than a month. If you have two programmers, X and Y, with X knowing Java but of mediocre programming talent and Y being a sharp programmer who does not yet know Java, Y will be running rings around X within a couple of weeks on the job, and Y will be the one who contributes the most to the timely and bug-free completion of the project.

The employers' policy is akin to saying an applicant for a job as a Chevy mechanic will be automatically rejected if his/her experience is with Fords. Again, the employers are harming themselves with such a policy. A short training period given to a competent programmer, would be far cheaper (not just in salary but also in the large bounty paid to an employment agent), and the employer would have a productive programmer on the project more quickly.

Among other things, current policy amounts to rampant age discrimination by employers. And it is vital to note that a mid-career programmer cannot solve the problem by retraining him/herself, say by taking a course in Java at the local community college. No employer would hire the programmer on that basis.

(ITAA claims there is a low unemployment rate among programmers. This is very misleading. If a mid-career programmer cannot find work as a programmer, he/she will take another kind of job such in marketing of software, or just leave the field entirely, and become, say, an insurance agent. Either way, these programmers would not show up in the unemployment data.)

On the ITAA survey's figure of 190,000 unfilled IT jobs:

This is very sloppy on DOC's part. Earlier in the report, DOC seemed to have implied that it was taking a much narrower definition of "IT worker" than did ITAA; if so, DOC should not quote ITAA's 190,000 figure without a disclaimer. Furthermore, DOC makes no mention of the methodological problems with ITAA's survey, which I believe were communicated to them.

On the "education pipeline":

I would first like to point out a ridiculous contradiction on this page: DOC dismisses operating systems as "computer theory," yet earlier on the same page DOC cited operating systems architects as a group getting the highest salary increases! And then on page 13 DOC says that systems analysts must be experts in operating systems!

These are not just amusing (or irritating) contradictions. They are symptomatic of the fact that DOC and ITAA simply don't know what they are talking about. They are not technical people, and thus are at the mercy of employers and their biases, and may not even accurately reproduce what employers tell them either.

(I will make further comments on "computer theory" later.)

Pages 11-12:

On production of computer science degrees:

First, there is a very serious error of omission here. Enrollment in university computer science majors exploded by 40% in 1996. This information was given to ITAA by the Computing Research Association (a consortium of university computer science departments) when ITAA was soliciting comments on the draft version of its report, but ITAA failed to include this data in their final report, and has not mentioned it in their numerous press interviews. This is an extremely dishonest, almost criminal, suppression of information on ITAA's part.

Second, though DOC at least does mention that one can be a programmer without a computer science degree, it does not recognize the fact that MOST programmers in fact do not have such degrees, so that the number of computer science graduates is not very important.

On enrollment of foreign students in graduate programs in computer science and engineering:

The data here are accurate but are EXTREMELY misleading. One does NOT need a graduate degree to be a programmer. Indeed, a joint study by Stanford and RAND found that we are greatly overproducing Ph.D.'s in engineering. The Stanford/RAND report found that the number of computer science Ph.D.'s produced was calculated on the departments' "needs" to produce Ph.D.'s, not on the basis of industrial needs. We simply do not need so many graduate students, foreign or domestic.

Foreign students join U.S. computer science graduate programs solely as a steppingstone to immigration. This was admitted by the National Association of Foreign Student Advisers (NAFSA) in 1995-1996, when they were lobbying against a congressional immigration bill. The bill at the time included a feature which would require foreign students to return to their home countries after finishing their studies in the U.S. NAFSA said, correctly, that the foreign students would not come to the U.S. in the first place if such a law were enacted, thus admitting that immigration, not education, is the students' motivation for coming here.

There are large numbers of foreign-national Asian students in U.S. gradaute programs in computer science, but very few Asian-American (U.S. citizen or permanent resident) students in those programs, in spite of the fact that there are large number of Asian-American students in the undergraduate curricula. Both groups place high cultural values on education, but the difference is due to the fact that for Asian foreign students there is an immigration incentive for graduate study, while the Asian-American students have no such incentive. For example, due to the large Asian-American population in California, at UC Davis Asian-American students comprise approximately 50% of the computer science enrollment at the undergraduate level, but only about 4% at the graduate level.

For the same reason, DOC's comment that U.S. employers cannot count on getting all, or most, of the graduate population because so many are foreign students is absurd. Virtually all of the foreign students stay in the U.S. and gain immigration status after graduation. In our Computer Science Department at UC Davis, for instance, we have had only two foreign graduate students return to their home countries (one to Hong Kong, the other to Austria) in the 17 years that I have been in the department. Again, we do not need these students anyway, but my point is that here again DOC shows that it simply does not know what it is talking about.

Page 16:

On the global "shortage":

Here DOC unconsciously admits that their earlier claims of a "shortage" was misleading. They quote the Deloitte and Touche survey as finding a shortage of programmers WITH SPECIFIC SOFTWARE SKILLS SUCH AS THE "SAP" PACKAGE. In other words, it is NOT the case that there is a shortage of "bodies," i.e. of professional programmers -- in stark contrast to the impression the reader gets by seeing DOC's repeated references to low numbers of graduates of computer science programs. (The DOC report also admits in more detail that skills, not bodies, is the issue, on page 24.) Again, the employers' obsession with specific skills is unwarranted and harmful to the success of their software projects, but my point here is that DOC and ITAA are wrong when they claim a shortage of "bodies."

As is true throughtout the DOC and ITAA reports, there is no statistical basis offered for the claim of a worldwide shortage.

Page 19:

On U.S. competitiveness in the IT industry:

It is really amazing that the IT industry, normally proponents of laissez-faire economics, suddenly wants federal government subsidies. I wonder if the industry would like our government to do what Japan's has. After years of pouring huge sums of money into its software industry, the Japanese government recently declared the program a failure, and officially classified the Japanese software sector as a "distressed industry."

The EDS example on this page is particularly disingenous. The firm claims that due to is long-term fixed contracts, it cannot pass salary increases on to customers, so it will have to reduce its work force. Oh really? What about those contracts? Is EDS just going to say, "We had to fire our workers, and thus can't deliver as promised in the contracts"? Of course not! Or is EDS just going to increase unpaid overtime to compensate? If so, aren't those programmers, whom DOC claims can find another job instantly due to the labor "shortage," just going to move to an employer who won't make them work evenings and weekends? The whole argument ITAA/DOC makes is a house of cards which collapses under the weight of its own contradictions.

Page 20:

On competitiveness, innovation and so on:

This is really absurd. With VERY rare exceptions (Japanese game software and the German SAP package), the U.S. has NO foreign competitors in software. Part of that is due to the increasingly monopolistic nature of the industry; even U.S. software companies have almost no hope of competing with Microsoft, let alone foreign companies.

The other major reason is the American innovative spirit. For instance the Japanese are masters at careful, reliable manufacturing of cars and electronics, but as mentioned earlier, they have utterly failed to develop a software industry. The same has happened in Taiwan, where the government has been funding a "Taiwanese Silicon Valley" at Hsinchu. The vast majority of the major technological advances in the computer industry have been made by Americans (in fact by American natives).

If the ITAA and DOC really want to keep the U.S. industry competitive and innovative, they should urge the industry to abandon its destructive hiring policies. Once word gets around that one can only have a 5- or 10-year career in software, how many of our "best and brightest" young university students will choose to major in computer science? Because of UC Davis' proximity to Silicon Valley, many of my students have parents who work there, a number of whom have either been laid off or who were working for startup companies which later went bankrupt. A number of these students have expressed concern to me for their own long-term prospects in the industry, and wondered if they should change their majors to something else. We of course should never promise them job security, but if the current situation continues, more and more of our bright young people will choose careers in more stable careers such as finance. Is that what ITAA and DOC really want? What would THAT do for U.S. competitiveness and innovation in the computer industry?

Page 23:

On the factors underlying a decline in production of degrees in computer science:

First, as stated earlier, there is a reprehensible error of omission here, as the ITAA (from which DOC got its information) suppressed information from the Computing Research Association that enrollment in computer science exploded by 40% in 1996. That big increase was clearly due to the growth of the industry which began a year or two earlier. The market does work!

Second, also as stated earlier, and is conceded by DOC, a computer science degree is not the only possible path to work in the programming field.

Third, DOC fails to recognize (and probably does not know) that enrollment in computer science curricula has been consistently strong and increasing in the 1990s in regions of concentration of the computer industry. For instance, in our Department of Computer Science at UC Davis, the enrollments (totals of enrollments in all courses) have shown the following trend:

1991-1992 2,254

1992-1993 2,600

1993-1994 3,305

1994-1995 3,485

1995-1996 3,731

1996-1997 (est.) 4,905

I believe that this has been due in large part to the fact that word of mouth has spread the news among students that many computer companies come to UC Davis to recruit new graduates for jobs. By contrast, I was told by the former chair of the Department of Computer Science at the University of Nebraska that most computer industry companies do NOT recruit at his school. What happens is that if a company does not find graduates in its particular region of the U.S., the company then recruits foreign nationals, making little or no effort to recruit graduates in other regions of the U.S. Students at the University of Nebraska do not see high degrees of recruiting activity in the computers area there, and thus choose other majors.

DOC correctly notes that due to reduction of the defense industry and corporate downsizing (as well as a number of other factors DOC seems to be unaware of), students in the early 1990s perceived that the number of job opportunities in the computer field had declined. The point is, then, that students at places like the University of Nebraska never heard the news that the situation had changed, until possibly the last year or so when a number of newspaper stories began to appear on the growth of the industry. Again, the point is that the market does work, and the industry ought to let it do so instead of making shameless requests for government subsidies.

On the math preparation of U.S. university students:

This is a red herring being pushed by ITAA, cleverly preying on Americans' fears that "Johnny can't do math."

First of all, this comprises yet another DOC contradiction. DOC says that math preparation has enjoyed "rapid GAINS in the last decade," and yet DOC says that this is a factor behind the DECLINE in production of computer science majors! How can GAINS of what is perceived by DOC as an essential quality for a product lead to DECLINES in that product????

Second, as I said, the whole thing is a red herring anyway. One does NOT use math as a programmer. Again, this is simply an effort to exploit an emotional issue which, though genuine, is unrelated to the subject at hand.

On the university computer science curricula:

I find DOC's (read "ITAA's") comments here very offensive.

The fact is that computer science curricula of the last 15 years have been quite practice-oriented. Typically only one or two theory courses are required for a computer science major, contrary to DOC's statement that the students are "superb theorists," The fact is that the student hate the small amount of theory we give them. If the course does not require programming, the students simply don't put much effort into the course, and the joke among the faculty in our department (which would resonate with most computer science faculty nationwide) is that our students consider themselves Computer Programming majors rather than Computer Science majors.

It is true that the curricula do not offer courses in every software technology under the sun. That would be impossible, as there are hundreds of different technologies, and we can't offer hundreds of courses. But again, we shouldn't! As I have said before, what really counts toward the success of a software project is having sharp programmers -- not sharp Oracle programmers, just sharp programmers, period, And indeed our computer science curricula are producing good programmers. A typical programming assignment will be more challenging than the typical assignment the student gets in industry after graduation, where programmers typically work on only small pieces of a large program.

By the way, we urge our students to get true real-world experience in summer jobs, internships and six-month co-op positions in the computer industry during their undergraduate careers, and student interest is extremely keen. There are far more interested students than jobs of this sort.

At any rate, this whole issue is yet another red herring. The IT industry loves to hire new graduates, and has stated publicly that it has deliberately changed its hiring focus toward new graduates in the last few years. Professor Edward Feigenbaum of the Stanford University Computer Science Department and an expert on the Japanese computer industry says, "Few if any authoritative Japanese are satisfied with the quality of the computer science and information science departments at Japan's universities. In the United States, I have never heard a single software industry executive claim to be unhappy with the output of the computer science departments of U.S. universities."

Page 24:

On retraining:

There are contradictions galore in this aspect of the ITAA and DOC statements.

ITAA and many individual employers have told the press (at different times) that "We DO retrain our people," and "We can't afford to train our people, because we always have very short deadlines." Come on, employers, you can't have it both ways!

When I interviewed a vice president of one of the most famous high-tech companies in the U.S. (who asked to speak on background), at first he gave me his standard statistics about how many millions of dollars his firm spends each year on training. But I asked him, "WHO is being trained here? Is it your engineers and programmers, or your technicians and secretaries?" He then admitted that it was the technicians and secretaries.

Most employers are not providing retraining for many of their programmers. The highest estimate I have seen of the percentage of computer programmers whose employers provide retraining is 20 to 25% from the Computerworld Salary and Job Satisfaction Survey.

As Kim Lee, of the Network Connections employment agency in the Silicon Valley remarked to me, "In 1988 the employers would have retrained [mid-career] people but they're not desperate enough to do so today." Note that this is yet another illustration of the fact that there is no software labor shortage.

As to the ITAA complaint that retraining an existing employee would just backfire, because the newly-"enfranchised" programmer could now jump to another company seeking those skills which would pay a higher salary, my answer is simple: If the industry did not have this ridiculous and unwarranted policy of overspecifying skills requirements, the above scenario could not occur in the first place.

(How to get employers to change this attitude is an interesting related question. A change would only work if they *all* changed, and the question then becomes how to ensure that all employers do change their attitudes. My answer to that is first to drastically reduce the yearly quota of work visas for foreign programmers; this would force employers to look more broadly at applicants for programming positions.)

What is most dishonest about all this talk from ITAA about retraining is that it is just a pretext. As mentioned earlier, even if a programmer retrains on his/her own, say by taking a Java course at a local community college, employers will NOT hire him/her on that basis. As Maryann Rousseau, also a computer industry employment agent, noted when I interviewed her, "Taking a course is just not going to work for a senior person, given his salary."

So it again boils down to a question of salary concerns among employers, especially when a large supply of foreign programmers is available. As mentioned before, the salaries paid to those programmers are on average substantially lower than what their domestic peers get, between 15 and 20%, according to my analysis of the Census data.

But even more significantly, even employers who pay "equal" wages to the foreign imports ARE STILL SAVING IN SALARY COSTS, for the following reason: As stated earlier, the industry has made a conscious effort to shift its hiring patterns to new collegegraduates, who are cheaper than mid-career people, and who are usually single and thus can put in large amounts of overtime without worrying about family obligations. So even an employer who pays "equal" wages to the foreign imports will still save on hiring costs by hiring them -- when the employer runs out of new American graduates to hire, he can hire the new young foreign graduates, instead of the more expensive domestic mid-career programmers.

Again, it is ironic, to say the least, that the notoriously laissez-faire computer industry wants government subsidies of various sorts, as indicated in the ITAA and DOC reports, with the biggest subsidy being the unspoken one of wage reductions enabled directly and indirectly by the supply of foreign labor. Why are they so afraid of letting the market determine the price of labor?

(By the way, q domestic programmer could not even offer to work for a lower salary, as he/she would never even be given an interview in the first place.)

Pages 25-26:

On the proposal to alleviate the "shortage" by recruiting more women and minorities into the profession:

This is nice, but again it is just a pretext on ITAA's part.

By the way, beware of the statistics here on Asians, which do not distinguish between Asian-American students (i.e. U.S. citizens and permanent residents) and Asian foreign students. The statistics here make no distinction at all, but even surveys which did make the distinction were found later to be inaccurate, as many reporting departments were counting Asian foreign students in the "Asian" category, missing the fact that the category was stated to mean Asian-American only.