Security Plan for UCLA-Mathnet

James F. Carter

2000-04-05

The Mathematics Department has done well in repelling the vast inimical forces on the Internet, and this plan will incorporate much of what we are already doing. However, it's helpful to have a written plan, for several reasons. First, we need to review periodically whether current threats match what we are defending against, and whether changes in the system may have reduced the effectiveness of our defenses. Second, it is easier to improve our defenses when we have the existing system available clearly for review.

1 Nature of Threats

What do we want to prevent hackers from doing? There are four basic areas:

1. Hiding data. Our users would prefer to share with others only the data which they have decided to share. We have an obligation under FERPA to keep private the information about who is a student and what classes they are taking. Perhaps a more serious problem is students trying to steal examinations before they are administered. But in the department the emphasis is more on sharing data, rather than hiding it.
2. Data integrity. The most likely threats in this area are probably where a student tries to change his grade in the instructor's online gradebook, or vandalism of web pages. Also in this area is sending disinformation by e-mail (e.g. an exam has been postponed) and attributing it to one of our professors.
3. Theft of services. In this category would be for unauthorized people to use software licensed solely for our use, or to use our computing power for projects not related to the department.
4. Hostile activities. Viruses generally use the infected machine to reproduce, imposing a considerable load on it. Spam is sent by misconfigured mail servers utilized by unauthorized third parties. Writeable FTP areas and virus-type FTP and IRC servers are used to share illegally copied files. Besides the loss of our resources, to avoid legal liability for complicity in the hostile activities, we need to exercise due diligence to prevent them.

Who are we protecting ourselves against? Most highly publicized attacks of the last several years have come from viruses, which pick victims at random, and whose main goal is to reproduce, leaving behind a payload of more or less subtle vandalism. Random attacks are also mounted by hands-on hackers, but though the attacks can be more sophisticated, the goals are pretty much the same; they use victim sites as transfer points to attack more victims. In contrast, if we have something valuable and someone knows we have it, he or she can make a targeted attack on us, and the hacker will be justified in devoting more resources in the effort than for a random attack. Particularly significant for us, our targets are mainly of interest to people (students) who have legitimate access to our machines, who legitimately know all details of our systems, and who have plenty of time on their hands.

A particularly troublesome aspect of this exercise, for me, is to get a feel for what the attackers can do to us. I discovered a web site, http://project.honeypot.org/. These people have set up a variety of instrumented networks of machines, with production operating systems and tools, to attract hackers. They then analyse what was done, and post the most interesting exploits in a lesson or contest type format, so sysadmins can learn how to deal with them. Here is a summary of these ``scans of the month'' since 2000.

• Looking for FTP and Front Page vulnerabilities.
• Looking for an exploitable debug tool in the cgi_bin directory.
• The ``ramen'' worm. Send a nasty string to a wu-ftpd server with the vulnerability, and get a root shell. Similarly to an unpatched rpc.statd.
• Identifying a Windows IIS server before attacking its vulnerabilities.
• Windows IIS Unicode directory traversal exploit.
• Another rpc.statd exploit succeeded.
• On Windows NT, exploiting the Unicode directory traversal bug, and an RDS (remote data service) exploit.
• Yet another rpc.statd exploit succeeded.

Many of the reported scenarios (not shown) are port scans, where the hacker looks for a vulnerable system. The successful exploits were all of vulnerabilities reported in CERT advisories. I have several conclusions:

• A stock, unpatched machine is hopeless. One of the test systems, running Red Hat 6.2, was broken into within 8 hours of being put on the net.
• A system with the latest patches is not vulnerable to the exploits generally used.
• When a new exploit is developed, someone has to be the first to be hacked with it, but with defense in depth and attention to patches, it's likely that we will not be the unlucky someone.
• It is not true that ``anyone can get root access on your system''. Vulnerabilities are quickly used, reported, and plugged. But so many systems are misconfigured, or hardly attended to at all, that successful exploits are very common.

But a question remains: is there a large class of exploits that can only be done by an interactive user, which would not be seen in this collection? I think not, but I cannot be sure.

2 What This Plan Skips

This is a plan for security of the computing network. I would like to mention up front some areas of security which are important, but which are outside the scope of this plan.

• Patches. We put a high priority on patching reported vulnerabilities both on Windows and UNIX. Only by doing this can we resist the inevitable attacks.
• Firmware. Vulnerabilities are regularly reported in printers and routers, and we are at the mercy of the vendors in getting patched firmware. These important devices should be regarded with grave suspicion.
• Backup. Disasters can be natural, e.g. earthquakes, or vandalism in the course of hacking. We keep multiple backup copies of user data so it can be restored, often to a point within 24 hours of loss, or in other cases within a week. We regularly access these backups, and we know they actually can be used. We ensure that the backups go away after a specified time, to limit the amount of work we have to do in response to a subpoena.
• Mail viruses. These generally work by inducing the user to execute code on a vulnerable machine. We intercept executable content delivered by our mail server (but not on outside servers downloaded by our users). We run virus scanning software on Windows machines (which sees mail from outside as well as inside), and we update the virus signatures regularly, not relying on user action. The default configuration for Windows mail readers does not automatically execute the content. (UNIX mail readers never execute messages.) But we need to make sure that the users do not voluntarily execute the content, and that they keep the virus software running.
• Permissions of data. The default permissions should be (and are) sufficient to protect sensitive data from disclosure or alteration. When opening up files for sharing, users have to be careful not to open it up too much. It might be a good idea for us to add a capability for user-managed groups, so the choice can be more than just ``secret'' vs. ``world access''. A capability for user or group authentication to receive web data would also be helpful.
• Mail identification. For many years the computing group has had a goal that all mail originating in the department should be cryptographically signed. But the available user agents have not been up to the task. Perhaps the most recent generation can be made to authoritatively identify senders, cleanly and by default.
• User authentication. In the past there were several nasty incidents where a user's account at another institution was compromised, and by remote execution privilege (.rhosts) the hacker got to us also. We need to insist on better inter-system authentication, for example by forbidding .rhosts and requiring ssh. We are pretty fascist about requiring passwords that should be hard to crack, but we could do better, for example by actually cracking passwords on a regular basis. We very much need to get rid of NIS for user authentication, because it makes encrypted passwords available to a locally active hacker even without root access, but our previous attempt ended in failure. This needs to be worked on again. The obvious candidate is Kerberos.
• Appropriate computing. Users need to exercise good judgment about what activities are appropriate for a State agency; for example, it would not be appropriate to maintain a mailing list for a political party. It is possible to annoy other users by visual imagery, for example a pornographic screen background. Users need to remember our privacy obligations; for example, TA's and instructors should not post students' names, registration numbers or grades on the web.

3 Ideal Configuration

The threats we mainly defend against are frequent random attacks from the global internet, and rare but sophisticated targeted attacks by insiders with interactive accounts, or who have compromised our user's account on another system. In some cases the hacker will be executing as root on one of our machines. Thus ``defense in depth'' is essential: multiple layers of defense, all of which the hacker has to work through. Mainly this plan describes the relations between different machines and subnets, so a sucessful exploit on one machine is contained there.

In the ideal configuration, we would have a perimeter network, to which we would connect a screening router (passing only selected traffic) to the PSnet-Internet, screening routers to each of our subnets, and one or more ``bastion hosts'', which would do most of the interacting with outsiders including hackers. Thus the bastion host is the most likely one to be damaged by enemy action. Preferably the perimeter net would be physically implemented rather than depending on Cisco VLANs. It could be carved out of a fraction of the 70 or 19 nets, similar to the two Beowulf clusters. Here is a diagram of such a network.

The bastion host(s) will serve all content that we intend to give to outsiders, such as web pages and anonymous FTP. Incoming e-mail (SMTP) will be accepted by the bastion host. In general, outsiders are not allowed to initiate connections to internal hosts, but internal hosts are allowed to initiate most connections to the outside. Users' home directories are on internal file servers and are not accessible from the bastion host. (This section describes an ideal design and it is understood that some security aspects may need to be relaxed due to operational requirements.)

Security flaws are discovered with discouraging regularity in the firmware of network printers. In this plan all the network printers are on a separate subnet, probably implemented as a VLAN. It would not be accessible at all from the Internet, but would be considered to be as untrustworthy as the Internet by internal hosts. Printers attached to actual computers would not be affected.

It's important to our users to be able to mount home and software directories from and to most internal machines. In an ideal system a strongly authenticated remote filesystem, such as AFS, would be used. Deployment of AFS is not very likely; instead, Sun's NFS, notorious for the weakness of its authentication, will probably continue to be used. NFS has its own host-based access control list, which would (and does) only allow internal machines to mount the files.

The following table lists the protocols and ports to be passed by the routers. Services not listed are blocked (until requested by users). In general, UDP ports for TCP-only services are blocked; however, UDP services often have TCP fallbacks which are restricted similarly to the UDP ports. These listings refer to packets that initiate connections; subsequent packets are generally between arbitrary ports, and have to be let through. ``Ext'' is the router from the perimeter net to PSnet and the Internet, ``Sub'' is a router between a client subnet and the perimeter net when the partner is external, and ``Int'' refers to the same routers communicating between our client nets through the perimeter net. Hosts which are capable (e.g. Linux) will individually implement the ``Int'' rules as well. ``I'' and ``O'' mean the connection is allowed inbound (Internet to local) or outbound. ``n'' means blocked in both directions. ``m'' means ``maybe''. ``TPB'' in the comments means ``a truly paranoid sysop would block this''.

Protocol	Ext	Sub	Int	Comments
ICMP	IO	IO	IO	Possibly selective by type
IPSec	IO	IO	IO	Also called ipv6-crypt
IPv6-auth	IO	IO	m
ospfigp	IO	IO	n	Cisco needs this?
mtp	I	I	IO	Multicast Transport Protocol
Port (TCP)	Ext	Sub	Int	Comments
ftp	IO	m	IO	TPB; sftp (ssh extension) could substitute
ssh	IO	IO	IO	ssh to home sites is allowed
telnet	n	n	n	We will try to insist on ssh
smtp	IO	IO	m	Mail exchange via Bastion
whois	O	O	n	TPB
domain	IO	IO	IO	This is DNS (53); UDP and TCP
bootp/s	n	n	n	To internal server only
tftp	n	n	n	To internal server only
gopher	O	O	n	TPB, presently useless
finger	IO	O	IO	Internet can only contact Bastion
http	IO	O	n	Internet can only contact Bastion
https	IO	O	n	Internet can only contact Bastion
kerberos	n	I	IO	Bastion relies on internal server
imap	IO	IO	IO	TPB; I wish we could block this
imaps (993)	IO	IO	IO	Secure imap is OK
pop3	IO	IO	IO	TPB; I wish we could block this
pop3s (995)	IO	IO	IO	Secure POP is OK
lpr (515)	m	O	IO	Net printers on separate subnet
ipp (631)	m	O	IO	For hosts with printers
sunrpc	n	I	IO	NFS filesystem mount, and other services
NFS (2049)	n	n	IO	UDP; internal hosts mount Bastion FS
ident/auth	n	n	n	Reject, not just drop packets
nntp	O	O	n	We have no news server
ntp	IO	IO	IO	UDP only
netbios-*	n	n	IO	Perimeter net has no SMB services
snmp	n	n	n	Encrypted v3 can be considered
irc	IO	IO	IO	Requires bidirectional connections
xdmcp	n	n	IO	X-Windows login
appletalk	n	n	m
ldap	O	O	O	Until we have our own LDAP server
ldaps	O	O	O	Until we have our own LDAP server
6000-6031	n	I	IO	X-Windows: require SSH port forwarding

4 Practical Configuration

In some cases security has to be sacrificed for useability. The major modifications we need to make are these. In this discussion it is assumed that a hacker is executing as root on the bastion host.

Fractional Subnet Perimeter:

At present, each subnet has a designated fraction (e.g. IP addresses from 240...254) which is ``outside the firewall'', subject to traffic restrictions as listed above for the external router, and all others are restricted more severely. The outside hosts are acting effectively as bastion hosts. The purpose is to allow users free access to their home sites from outside, and thus we are exposing our most valuable machines to hacking. We need to make the political effort to educate our users how ``free access'' can be set up to be both better and more secure, e.g. with sftp or slogin; the goal is to put the home sites under maximal restriction, not minimal.

Internal NFS Mounting:

In general, a user expects to get access to his/her UNIX and Windows home directories from any machine in the department. Non-system software (e.g. Matlab) is served from a crossmounted directory, both on UNIX and Windows. But the root partition and basic system software are not accessible across the net. In the past there was a host-based restriction against crossmounting PIC and Math, but numeric user IDs are maintained carefully so there are no duplications, and this restriction has been relaxed in the past few years. Since the bastion host is most likely to be hacked, it would be a good idea if it could not mount users' home directories.

We would really like to use a more secure inter-host filesystem mounting scheme, such as AFS. AFS also has the advantage that readonly volumes (for software) can be replicated with transparent failover when a server goes down; writeable (home directory) volumes get a local write-through cache, and files can be exported reasonably safely to alien hosts.

Mounting Web Pages:

Our users expect to be able to create web pages for consumption over the Internet. Presently they go in $HOME/public_html, which is exported by NFS to the web server. Users would resist if they had to go through a high security interface to publish web pages. One possibility is to export the home directory filesystems (readonly) to the web server on the bastion host. This is actually done now, except access is read-write. If that were done, the hacker could read any user's file. If specifically the public_html directories were exported, the hacker could only see material intended for public consumption (plus .htaccess and .htpasswd files), but the administrative work would be greater. Perhaps a better solution would be for the bastion host to export (read-write) to the internal hosts a filesystem containing web directories, one per user named for and owned by the user. To preserve the illusion of local access, the user could put a symbolic link to it in his home directory. A similar arrangement could be (and actually already is) implemented for material to be published via anonymous FTP.

CGI Execution:

This refers to executing a program on the web server in conjunction with showing a web page or responding to data on a web form. A number of users have ``vanity'' CGI scripts of low value such as hit counters, but a few have real operational needs for CGI, such as mathematical or statistical tools and demo software. But such software is often written with naive assumptions about security. The threat is that nasty input could induce the CGI script to execute arbitrary commands, as if the hacker had an interactive session. I believe that the SUexec mechanism is strong enough to ensure that such execution occurs as the user, so that only the user's own pages could be defaced. Another problem is that execution is on the web server (bastion host) itself, which gets overloaded when the application is computationally intensive or goes into an infinite loop. In PIC 40 the students learn to write CGI scripts, so the capability has to be supported on the server.

Remote NFS Mounting:

In the past we have had requests from politically influential users to let them mount their Mathnet home directories on a non-Mathnet system. The exported users were relegated to a separate machine and filesystem, for damage control in case of possible hacking. We should vigorously resist any future such requests. We should be pro-active and develop the tools to support remote mounting securely, such as AFS, and we should provide the client tools for our users to mount alien filesystems locally.

Remote SMB Mounting:

In PIC it has been the policy to allow students, at least those in the dormitories (on campus), to mount their PIC home directories on their home computers. Since the dorms are a hotbed of both virus activity and motivated targeted attackers, this policy should be terminated. A working and safe substitute should be provided. We presently have a FTP server product, and we let students deposit their homework, connect to the server via telnet, and compile their programs. An alternative might be for the users to execute the sftp component of ssh on a PIC UNIX machine, and access the home directory via Samba. This provides a nice drag and drop interface, and so far as we know it is reasonably resistant to nastiness sent down the connection. An AFS client for Windows NT is available.

Remote Printing:

Network printers cannot be trusted to resist attacks, and should not be allowed any contact with the Internet. If we decide to allow printing from alien sites -- and I believe this is an operational requirement for delivering hardcopy confirmations of financial transactions from Murphy Hall -- BSD style print requests should be directed to the bastion host, which would forward the data to the actual printer. It is possible to enumerate the sending sites on a special case basis.

Realtime Collaboration Tools:

Users (faculty, staff and students) need to be able to use the computer system to collaborate; here are a few application categories:

• Instant Messenger: For ``talking'' to colleagues in real time, and keeping track of whether they are available online.
• Calendar Management: To display and alter the times of someone's appointments. Only the user or a designated secretary should be able to alter the calendar.
• Group Authorship or ``Net Meeting'': Preparing a document with each author doing his or her part. The work in progress is visible at all times. It is particularly useful as a written record of a discussion.
• Discussion Lists: The same kind of thing but the participants post unitary messages, which are answered by similar postings.

There is software implementing each of these tools, and in general the software is notoriously insecure. We need to be pro-active in providing secure variants or alternative implementations of these functions for our users, and particularly, we need to resist demands to put the insecure versions on our system or let their traffic through our firewall.

Mail Delivery.

On UNIX it involves running arbitrary user-provided delivery agents and writing into arbitrary users' files, and thus it requires a process (sendmail) running as root to ``su'' to the actual user. Here is a potential for abuse, but since the defeat of the Morris Worm, exploits against sendmail have been rare. Our mail model would be that incoming mail goes to the bastion host which relays it to an internal host, and vice versa; internal hosts would no longer do SMTP to the global Internet.

Mailbox_Access.

We would, however, like to be more paranoid about the mailboxes themselves. Placement on the bastion host is not secure because we need to safeguard mail same as other home directory content, and also, we don't want spam filters or other mail management software to execute as the user and mount the home directory on the bastion host. Thus the mailbox has to be on the home server, preferably in the home directory itself. We are not happy with the security of the IMAP and POP3 protocols for moving mail to the user agent (reader). Secure (encrypted) versions of both protocols exist, and we should deploy the daemons, and should encourge or require that they be used. Some user agents such as pine can use ssh to execute a remote IMAP server specific to the owning user. If all reasonable user agents can do this, we should consider blocking IMAP and POP3 at the firewall, or even decommitting the server daemons for them.

5 Recommendations

1. Define with our user community the types of service which have to function through our firewall, and particularly, get rid of dispensible services like telnet and remote SMB (Windows) mounting.
2. Educate our users about more secure and functionally better solutions to their problems, such as ssh for remote login and sftp for file transfer. Make sure sftp works for Windows directories. Investigate secure mailbox access.
3. Move outward-facing services, like mail, web and FTP, to one or at most two bastion hosts. In part, this is already done.
4. Move users' UNIX home machines, and all Windows machines, inside the firewall. Organize the subnets for special protection of, and against, network printers. A physically implemented perimeter net could be helpful here.
5. Obtain and install the more secure variants of modern ``groupware'' and educate our users to get the most benefit from them.
6. Improve our presently inadequate and insecure user authentication procedures (e.g. NIS replaced by Kerberos) and inter-host filesystem export (NFS replaced by AFS).