Syngress Hack The Stack Using Snort And Ethereal To Master The 8 Layers Of An Insecure Network Jan 2007 ISBN 1597491098 pdf

  Visit us at w w w . s y n g r e s s . c o m

  Syngress is committed to publishing high-quality books for IT Professionals and delivering those books in media and formats that fit the demands of our cus- tomers. We are also committed to extending the utility of the book you purchase via additional materials available from our Web site.

  SOLUTIONS WEB SITE

  To register your book, visit www.syngress.com/solutions. Once registered, you can access our solutions@syngress.com Web pages. There you will find an assortment of value-added features such as free e-books related to the topic of this book, URLs of related Web site, FAQs from the book, corrections, and any updates from the author(s).

  ULTIMATE CDs

  Our Ultimate CD product line offers our readers budget-conscious compilations of some of our best-selling backlist titles in Adobe PDF form. These CDs are the perfect way to extend your reference library on key topics pertaining to your area of exper- tise, including Cisco Engineering, Microsoft Windows System Administration, CyberCrime Investigation, Open Source Security, and Firewall Configuration, to name a few.

  DOWNLOADABLE E-BOOKS

  For readers who can’t wait for hard copy, we offer most of our titles in download- able Adobe PDF form. These eBooks are often available weeks before hard copies, and are priced affordably.

  SYNGRESS OUTLET

  Our outlet store at syngress.com features overstocked, out-of-print, or slightly hurt books at significant savings.

  SITE LICENSING

  Syngress has a well-established program for site licensing our e-books onto servers in corporations, educational institutions, and large organizations. Contact us at sales@syngress.com for more information.

  CUSTOM PUBLISHING

  Many organizations welcome the ability to combine parts of multiple Syngress books, as well as their own content, into a single volume for their own internal use. Contact us at sales@syngress.com for more information.

  Hack the Stack

  

U S I N G S N O R T A N D E T H E R E A L T O M A S T E R

T H E 8 L AY E R S O F A N I N S E C U R E N E T W O R K

  Hack the Stack

  

U S I N G S N O R T A N D E T H E R E A L T O M A S T E R

T H E 8 L AY E R S O F A N I N S E C U R E N E T W O R K

Michael Gregg Technical Editor Stephen Watkins George Mays

  

Syngress Publishing, Inc., the author(s), and any person or firm involved in the writing, editing, or produc-

tion (collectively “Makers”) of this book (“the Work”) do not guarantee or warrant the results to be obtained from the Work.

There is no guarantee of any kind, expressed or implied, regarding the Work or its contents.The Work is

sold AS IS and WITHOUT WARRANTY.You may have other legal rights, which vary from state to state. In no event will Makers be liable to you for damages, including any loss of profits, lost savings, or other

incidental or consequential damages arising out from the Work or its contents. Because some states do not

allow the exclusion or limitation of liability for consequential or incidental damages, the above limitation

may not apply to you.

  

You should always use reasonable care, including backup and other appropriate precautions, when working

with computers, networks, data, and files. Syngress Media®, Syngress®, “Career Advancement Through Skill Enhancement®,” “Ask the Author

UPDATE®,” and “Hack Proofing®,” are registered trademarks of Syngress Publishing, Inc. “Syngress:The

Definition of a Serious Security Library” ™, “Mission Critical™,” and “The Only Way to Stop a Hacker is to Think Like One ™” are trademarks of Syngress Publishing, Inc. Brands and product names mentioned in this book are trademarks or service marks of their respective companies.

KEY SERIAL NUMBER

  001 HJIRTCV764 002 PO9873D5FG 003 829KM8NJH2 004 HEATHTANER 005 CVPLQ6WQ23 006

  VBP965T5T5 007 HJJJ863WD3E 008 2987GVTWMK 009 629MP5SDJT 010

  IMWQ295T6T PUBLISHED BY Syngress Publishing, Inc. 800 Hingham Street Rockland, MA 02370 Hack the Stack: Using Snort and Ethereal to Master the 8 Layers of an Insecure Netork Copyright © 2006 by Syngress Publishing, Inc. All rights reserved. Printed in the United States of

America. Except as permitted under the Copyright Act of 1976, no part of this publication may be repro-

duced or distributed in any form or by any means, or stored in a database or retrieval system, without the

prior written permission of the publisher, with the exception that the program listings may be entered, stored, and executed in a computer system, but they may not be reproduced for publication. Printed in the Canada 1 2 3 4 5 6 7 8 9 0

  ISBN: 1-59749-109-8 Publisher: Andrew Williams Page Layout and Art: Patricia Lupien Acquisitions Editor: Gary Byrne Copy Editor: Judy Eby Technical Editor: Stephen Watkins Indexer: Odessa&Cie

  Acknowledgments Syngress would like to acknowledge the following people for their kindness and support in making this book possible. Syngress books are now distributed in the United States and Canada by

O’Reilly Media, Inc.The enthusiasm and work ethic at O’Reilly are incredible,

and we would like to thank everyone there for their time and efforts to bring

Syngress books to market:Tim O’Reilly, Laura Baldwin, Mark Brokering, Mike

Leonard, Donna Selenko, Bonnie Sheehan, Cindy Davis, Grant Kikkert, Opol Matsutaro, Steve Hazelwood, Mark Wilson, Rick Brown,Tim Hinton, Kyle

Hart, Sara Winge, Peter Pardo, Leslie Crandell, Regina Aggio Wilkinson, Pascal

Honscher, Preston Paull, Susan Thompson, Bruce Stewart, Laura Schmier, Sue Willing, Mark Jacobsen, Betsy Waliszewski, Kathryn Barrett, John Chodacki, Rob Bullington, Kerry Beck, Karen Montgomery, and Patrick Dirden. The incredibly hardworking team at Elsevier Science, including Jonathan Bunkell, Ian Seager, Duncan Enright, David Burton, Rosanna Ramacciotti, Robert Fairbrother, Miguel Sanchez, Klaus Beran, Emma Wyatt, Krista Leppiko, Marcel Koppes, Judy Chappell, Radek Janousek, Rosie Moss, David Lockley, Nicola Haden, Bill Kennedy, Martina Morris, Kai Wuerfl-Davidek, Christiane Leipersberger,Yvonne Grueneklee, Nadia Balavoine, and Chris Reinders for making certain that our vision remains worldwide in scope.

David Buckland, Marie Chieng, Lucy Chong, Leslie Lim, Audrey Gan, Pang Ai

Hua, Joseph Chan, June Lim, and Siti Zuraidah Ahmad of Pansing Distributors

for the enthusiasm with which they receive our books.

David Scott, Tricia Wilden, Marilla Burgess, Annette Scott, Andrew Swaffer,

Stephen O’Donoghue, Bec Lowe, Mark Langley, and Anyo Geddes of Woodslane

for distributing our books throughout Australia, New Zealand, Papua New

Guinea, Fiji,Tonga, Solomon Islands, and the Cook Islands.

  Lead Author Michael Gregg is the President of Superior Solutions, Inc. and has more than 20 years’ experience in the IT field. He holds two associate’s degrees, a bachelor’s degree, and a master’s degree and is certified as CISSP, MCSE, MCT, CTT+, A+, N+, Security+, CNA, CCNA, CIW Security Analyst, CCE,

CEH, CHFI, CEI, DCNP, ES Dragon IDS, ES Advanced

Dragon IDS, and TICSA.

  Michael’s primary duties are to serve as project lead for

security assessments helping businesses and state agencies

secure their IT resources and assets. Michael has authored four books, including Inside Network Security Assessment, CISSP Prep Questions, CISSP Exam Cram2, and Certified Ethical Hacker Exam Prep2. He has developed four high-level security classes, including Global Knowledge’s Advanced Security Boot Camp,

Intense School’s Professional Hacking Lab Guide, ASPE’s

Network Security Essentials, and Assessing Network Vulnerabilities. He has created over 50 articles featured in mag- azines and Web sites, including Certification Magazine, GoCertify, The El Paso Times, and SearchSecurity.

  Michael is also a faculty member of Villanova University

and creator of Villanova’s college-level security classes,

including Essentials of IS Security, Mastering IS Security, and Advanced Security Management. He also serves as a site expert

for four TechTarget sites, including SearchNetworking,

SearchSecurity, SearchMobileNetworking, and SearchSmallBiz. He is a member of the TechTarget Editorial Board.

  Contributing Authors Ronald T. Bandes (CISSP, CCNA, MCSE, Security+) is an independent security consultant. Before becoming an indepen- dent consultant, he performed security duties for Fortune 100 companies such as JP Morgan, Dun and Bradstreet, and EDS. Ron holds a B.A. in Computer Science.

  Brandon Franklin (GCIA, MCSA, Security+) is a network administrator with KIT Solutions. KIT Solutions, Inc. (KIT stands for Knowledge Based Information Technology) creates intelligent systems for the health and human services industry that monitor and measure impact and performance outcomes and provides knowledge for improved decision making. A KIT system enables policy makers, government agencies, private foundations, researchers, and field practitioners to implement

best practices and science-based programs, demonstrate

impacts, and continuously improve outcomes.

  Brandon formerly served as the Team Lead of Intrusion Analysis at VigilantMinds, a Pittsburgh-based managed security services provider.

  Brandon cowrote Chapter 3 and wrote Chapter 6.

  George Mays (CISSP, CCNA, A+, Network+, Security+, I-

Net+) is an independent consultant who has 35 years’ experi-

ence in computing, data communications, and network security. He holds a B.S. in Systems Analysis. He is a member of the IEEE, CompTIA, and Internet Society.

  

Chris Ries is a Security Research Engineer for VigilantMinds

Inc., a managed security services provider and professional consulting organization based in Pittsburgh. His research

focuses on the discovery, exploitation, and remediation of soft-

ware vulnerabilities, analysis of malicious code, and evaluation

of security software. Chris has published a number of advi-

sories and technical whitepapers based on his research and has

contributed to several books on information security.

  Chris holds a bachelor’s degree in Computer Science with a Mathematics Minor from Colby College, where he com-

pleted research involving automated malicious code detection.

Chris has also worked as an analyst at the National Cyber- Forensics & Training Alliance (NCFTA) where he conducted technical research to support law enforcement.

  Chris wrote Chapter 8.

  Technical Editor Stephen Watkins (CISSP) is an Information Security Professional with more than 10 years of relevant technology experience, devoting eight of these years to the security field. He currently serves as Information Assurance Analyst at Regent University in southeastern Virginia. Before coming to Regent, he led a team of security professionals providing in- depth analysis for a global-scale government network. Over the last eight years, he has cultivated his expertise with regard to perimeter security and multilevel security architecture. His Check Point experience dates back to 1998 with FireWall-1 version 3.0b. He has earned his B.S. in Computer Science from Old Dominion University and M.S. in Computer Science, with Concentration in Infosec, from James Madison University. He is nearly a life-long resident of Virginia Beach, where he and his family remain active in their Church and the local Little League.

  Stephen wrote Chapter 7.

  

Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

Chapter 1 Extending OSI to Network Security . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Our Approach to This Book . . . . . . . . . . . . . . . . . . . . . . . . .2 Tools of the Trade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Protocol Analyzers . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Intrusion Detection Systems . . . . . . . . . . . . . . . . . . . .3 Organization of This Book . . . . . . . . . . . . . . . . . . . . . . .4 The People Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 The Application Layer . . . . . . . . . . . . . . . . . . . . . . . .6 The Presentation Layer . . . . . . . . . . . . . . . . . . . . . . . .6 The Session Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 The Transport Layer . . . . . . . . . . . . . . . . . . . . . . . . .6 The Network Layer . . . . . . . . . . . . . . . . . . . . . . . . . .7 The Data Link Layer . . . . . . . . . . . . . . . . . . . . . . . . .7 The Physical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Common Stack Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 The People Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 The Application Layer . . . . . . . . . . . . . . . . . . . . . . . . . . .8 The Session Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 The Transport Layer . . . . . . . . . . . . . . . . . . . . . . . . . . .10 The Data Link Layer . . . . . . . . . . . . . . . . . . . . . . . . . .11 The Physical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Mapping OSI to TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . .13 Countermeasures Found in Each Layer . . . . . . . . . . . . .14 The Current State of IT Security . . . . . . . . . . . . . . . . . . . .16 Physical Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Contents

  xiv Contents Signal Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Computer Security . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Information Security . . . . . . . . . . . . . . . . . . . . . . . . . . .19

  Using the Information in This Book . . . . . . . . . . . . . . . . . .19 Vulnerability Testing . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Security Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Finding and Reporting Vulnerabilities . . . . . . . . . . . . . .21

  Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . .25

Chapter 2 The Physical Layer . . . . . . . . . . . . . . . . . . . . . 27 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Defending the Physical Layer . . . . . . . . . . . . . . . . . . . . . . . .28 Design Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Perimeter Security . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Gates, Guards, and Grounds Design . . . . . . . . . . . . . .32 Facility Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Entry Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 Device Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 Identification and Authentication . . . . . . . . . . . . . . .39 Computer Controls . . . . . . . . . . . . . . . . . . . . . . . . .41 Mobile Devices and Media . . . . . . . . . . . . . . . . . . . .41 Communications Security . . . . . . . . . . . . . . . . . . . . . . .44 Bluetooth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44

  802.11 Wireless Protocols . . . . . . . . . . . . . . . . . . . . .46 Attacking the Physical Layer . . . . . . . . . . . . . . . . . . . . . . . .47 Stealing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 Data Slurping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 Lock Picks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 Wiretapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 Scanning and Sniffing . . . . . . . . . . . . . . . . . . . . . . . . . .54 The Early History of Scanning and Sniffing . . . . . . . .54

  Contents xv

Hardware Hacking . . . . . . . . . . . . . . . . . . . . . . . . . . . .57

Bypassing Physical Controls . . . . . . . . . . . . . . . . . . .58

  Modifying Hardware . . . . . . . . . . . . . . . . . . . . . . . .59

Layer 1 Security Project . . . . . . . . . . . . . . . . . . . . . . . . . . .64

One-Way Data Cable . . . . . . . . . . . . . . . . . . . . . . . . . .64

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65

  

Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66

Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . .67

Chapter 3 Layer 2: The Data Link Layer. . . . . . . . . . . . . 69 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 Ethernet and the Data Link Layer . . . . . . . . . . . . . . . . . . . .70 The Ethernet Frame Structure . . . . . . . . . . . . . . . . . . . .71 Understanding MAC Addressing . . . . . . . . . . . . . . . . . .72 Identifying Vendor Information . . . . . . . . . . . . . . . . .72 Performing Broadcast and Multicast . . . . . . . . . . . . .73 Examining the EtherType . . . . . . . . . . . . . . . . . . . . . . .73 Understanding PPP and SLIP . . . . . . . . . . . . . . . . . . . . . . .73 Examining SLIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 Examining PPP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74 Working with a Protocol Analyzer . . . . . . . . . . . . . . . . . . . .75 Writing BPFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77 Examining Live Traffic . . . . . . . . . . . . . . . . . . . . . . . . . .78 Filtering Traffic, Part Two . . . . . . . . . . . . . . . . . . . . . . . .79 Understanding How ARP Works . . . . . . . . . . . . . . . . . . . .82 Examining ARP Packet Structure . . . . . . . . . . . . . . . . .82 Attacking the Data Link Layer . . . . . . . . . . . . . . . . . . . . . . .84 Passive versus Active Sniffing . . . . . . . . . . . . . . . . . . . . .85 ARP Poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 ARP Flooding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 Routing Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 Sniffing Wireless . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88 Netstumbler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88 Kismet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88 Cracking WEP . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89 Wireless Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . .90

  xvi Contents MITM Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 Defending the Data Link Layer . . . . . . . . . . . . . . . . . . . . . .91

  Securing Your Network from Sniffers . . . . . . . . . . . . . . . . .91 Using Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 Secure Shell (SSH) . . . . . . . . . . . . . . . . . . . . . . . . . .92 Secure Sockets Layers (SSL) . . . . . . . . . . . . . . . . . . . . .92

  PGP and S/MIME . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 Employing Detection Techniques . . . . . . . . . . . . . . . . . . . . .93 Local Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

  Network Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 DNS Lookups . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 Driver Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 Network Monitor . . . . . . . . . . . . . . . . . . . . . . . . . .95 Using Honeytokens . . . . . . . . . . . . . . . . . . . . . . . . .95

  Data Link Layer Security Project . . . . . . . . . . . . . . . . . . . . .95 Using the Auditor Security Collection to Crack WEP . . . . .95 Cracking WEP with the Aircrack Suite . . . . . . . . . . .96 Cracking WPA with CoWPAtty . . . . . . . . . . . . . . . .98

  Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .101

Chapter 4 Layer 3: The Network Layer . . . . . . . . . . . . 103 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .104 The IP Packet Structure . . . . . . . . . . . . . . . . . . . . . . . . . .104 Identifying IP’s Version . . . . . . . . . . . . . . . . . . . . . . . .106 Type of Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107 Total Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .110 Datagram ID Number . . . . . . . . . . . . . . . . . . . . . . . . .110 Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111 Time to Live (TTL) . . . . . . . . . . . . . . . . . . . . . . . . . .112 Protocol Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115 Checksum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116 IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116

  Contents xvii The ICMP Packet Structure . . . . . . . . . . . . . . . . . . . . . . .118

  ICMP Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118

  ICMP Message Types and Format . . . . . . . . . . . . . . . .118 Common ICMP Messages . . . . . . . . . . . . . . . . . . . . . .119

Destination Unreachable . . . . . . . . . . . . . . . . . . . . .120

Traceroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121

Path MTU Discovery . . . . . . . . . . . . . . . . . . . . . .122

Redirects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122

  Attacking the Network Layer . . . . . . . . . . . . . . . . . . . . . .123

  IP Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124

Spoofing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124

Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . .124

Passive Fingerprinting . . . . . . . . . . . . . . . . . . . . . . .126

p0f—a Passive Fingerprinting Tool . . . . . . . . . . . . . .129

  

IP’s Role in Port Scanning . . . . . . . . . . . . . . . . . . .131

  ICMP Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133

Covert Channels . . . . . . . . . . . . . . . . . . . . . . . . . . .133

  

ICMP Echo Attacks . . . . . . . . . . . . . . . . . . . . . . . .136

Port Scanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136

OS Fingerprinting . . . . . . . . . . . . . . . . . . . . . . . . .137

DoS Attacks and Redirects . . . . . . . . . . . . . . . . . . .137

  Router and Routing Attacks . . . . . . . . . . . . . . . . . . . .138

Network Spoofing . . . . . . . . . . . . . . . . . . . . . . . . .139

Defending the Network Layer . . . . . . . . . . . . . . . . . . . . . .140 Securing IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .140

  Securing ICMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .140 Securing Routers and Routing Protocols . . . . . . . . . . .141

Address Spoofing . . . . . . . . . . . . . . . . . . . . . . . . . .142

Network Layer Security Project . . . . . . . . . . . . . . . . . . . . .143

  Ptunnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143 ACKCMD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .149

  xviii Contents

Chapter 5 Layer 4: The Transport Layer. . . . . . . . . . . . 151 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152 Connection-Oriented versus Connectionless Protocols . . . .152 Connection-Oriented Protocols . . . . . . . . . . . . . . . . . .152 Connectionless Protocols . . . . . . . . . . . . . . . . . . . . . . .153 Why Have Both Kinds of Protocols? . . . . . . . . . . . . . .153 Protocols at the Transport Layer . . . . . . . . . . . . . . . . . . . . .153 UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154 TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155 Source and Destination Ports . . . . . . . . . . . . . . . . . .156 Source Sequence Number

  and Acknowledgment Sequence Number . . . . . . . .157 Data Offset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158 Control Bits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158 Window Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159 Checksum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159 Urgent Pointer . . . . . . . . . . . . . . . . . . . . . . . . . . . .160

  How TCP Sessions Begin and End . . . . . . . . . . . . . . . .160 TCP Session Startup . . . . . . . . . . . . . . . . . . . . . . . .160 TCP Session Teardown . . . . . . . . . . . . . . . . . . . . . .161 The Hacker’s Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .162

  Some Common Attacks . . . . . . . . . . . . . . . . . . . . . . . .163 Scanning the Network . . . . . . . . . . . . . . . . . . . . . . . . . . .163 Port Scanning Overview . . . . . . . . . . . . . . . . . . . . . . .164 TCP Scan Variations . . . . . . . . . . . . . . . . . . . . . . . . . .165 Nmap Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165 Nmap:The Most Well Known Scanning Tool . . . . . .167

  Amap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .170 Scanrand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172 Operating System Fingerprinting . . . . . . . . . . . . . . . . . . . .173 How OS Discovery Works . . . . . . . . . . . . . . . . . . . . . .174

  Xprobe2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .176 OS Fingerprinting with Nmap . . . . . . . . . . . . . . . . . .179 Detecting Scans on Your Network . . . . . . . . . . . . . . . . . . .181 Snort Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182

  Contents xix The Snort User Interface— Basic Analysis and Security Engine . . . . . . . . . . . . .182

  

Defending the Transport Layer . . . . . . . . . . . . . . . . . . . . . .183

How the SSL Protocol Operates . . . . . . . . . . . . . . . . .184

Phase 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184 Phase 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185 Phase 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185

How SSL Appears on the Network . . . . . . . . . . . . . . .185

  

SSL/TLS Summary . . . . . . . . . . . . . . . . . . . . . . . . . . .187

Transport Layer Project—Setting Up Snort . . . . . . . . . . . .187

Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .188

Install Fedora Core 4 . . . . . . . . . . . . . . . . . . . . . . . . . .188

Install Supporting Software . . . . . . . . . . . . . . . . . . . . .190

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .200

  

Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .200

Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .202

Chapter 6 Layer 5: The Session Layer . . . . . . . . . . . . . 205 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .206 Attacking the Session Layer . . . . . . . . . . . . . . . . . . . . . . . .206 Observing a SYN Attack . . . . . . . . . . . . . . . . . . . . . . .206 Session Hijacking . . . . . . . . . . . . . . . . . . . . . . . . . . . .209 Session Hijacking Tools . . . . . . . . . . . . . . . . . . . . . .213 Domain Name System (DNS) Poisoning . . . . . . . . .216 Sniffing the Session Startup . . . . . . . . . . . . . . . . . . . . .218 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .219 Authenticating with Password Authentication Protocol . . . . . . . . . . . . . . . . . . . . .219 Authenticating with the Challenge Handshake Authentication Protocol . . . . . . . . . . . . . . . . . . . . .219 Authenticating with Local Area Network Manager and NT LAN Manager . . .220 Authenticating with NTLMv2 . . . . . . . . . . . . . . . .220 Authenticating with Kerberos . . . . . . . . . . . . . . . . .220 Tools Used for Sniffing the Session Startup . . . . . . .221 Observing a RST Attack . . . . . . . . . . . . . . . . . . . . . . .223

  xx Contents Defending the Session Layer . . . . . . . . . . . . . . . . . . . . . . .227 Mitigating DoS Attacks . . . . . . . . . . . . . . . . . . . . . . . .227

  Preventing Session Hijacking . . . . . . . . . . . . . . . . . . . .228 Selecting Authentication Protocols . . . . . . . . . . . . . . . .229 Defending Against RST Attacks . . . . . . . . . . . . . . . . . .231 Detecting Session Layer Attacks . . . . . . . . . . . . . . . . . .232 Port Knocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232 Session Layer Security Project . . . . . . . . . . . . . . . . . . . . . .232 Using Snort to Detect Malicious Traffic . . . . . . . . . . . .233 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .237

  Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .237 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .239

Chapter 7 Layer 6: The Presentation Layer . . . . . . . . . 241 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .242 The Structure of NetBIOS and SMB . . . . . . . . . . . . . . . . .242 Attacking the Presentation Layer . . . . . . . . . . . . . . . . . . . .245 NetBIOS and Enumeration . . . . . . . . . . . . . . . . . . . . .245 Exploiting the IPC$ Share . . . . . . . . . . . . . . . . . . .247 Sniffing Encrypted Traffic . . . . . . . . . . . . . . . . . . . . . .250 Attacking Kerberos . . . . . . . . . . . . . . . . . . . . . . . . .253 Tools to Intercept Traffic . . . . . . . . . . . . . . . . . . . . .257 Defending the Presentation Layer . . . . . . . . . . . . . . . . . . .266 Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .266 The Role of IPSec . . . . . . . . . . . . . . . . . . . . . . . . . . .268 Protecting E-mail . . . . . . . . . . . . . . . . . . . . . . . . . . . .272 Secure/Multipurpose Internet Mail Extensions . . . .272 Tightening NetBIOS Protections . . . . . . . . . . . . . . . . .273 Presentation Layer Security Project . . . . . . . . . . . . . . . . . .274 Subverting Encryption and Authentication . . . . . . . . . .274 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .280 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .280 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .282 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .283

  Contents xxi

Chapter 8 Layer 7: The Application Layer . . . . . . . . . . 285 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .286 The Structure of FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . .286 FTP Protocol Overview . . . . . . . . . . . . . . . . . . . . . . .286 FTP Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .288 FTP Security Issues . . . . . . . . . . . . . . . . . . . . . . . . . . .291 Analyzing Domain Name System and Its Weaknesses . . . . .292 DNS Message Format . . . . . . . . . . . . . . . . . . . . . . . . .292 The DNS Lookup Process . . . . . . . . . . . . . . . . . . . . . .295 The DNS Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . .296 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .296 Zones and Zone Transfers . . . . . . . . . . . . . . . . . . . . . .297 DNS Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .297 DNS Security Issues . . . . . . . . . . . . . . . . . . . . . . . . . .298 Other Insecure Application Layer Protocols . . . . . . . . . . . .299 Simple Mail Transfer Protocol . . . . . . . . . . . . . . . . . . .299 SMTP Protocol Overview . . . . . . . . . . . . . . . . . . .299 SMTP Security Issues . . . . . . . . . . . . . . . . . . . . . . .300 Telnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .301 Protocol Overview . . . . . . . . . . . . . . . . . . . . . . . . .302 Security Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . .302 Other Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302 Attacking the Application Layer . . . . . . . . . . . . . . . . . . . . .303 Attacking Web Applications . . . . . . . . . . . . . . . . . . . . .303 SQL Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . .303 Code Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . .304 Cross-Site Scripting . . . . . . . . . . . . . . . . . . . . . . . .305 Directory Traversal Attacks . . . . . . . . . . . . . . . . . . .307 Information Disclosure . . . . . . . . . . . . . . . . . . . . . .307 Authentication and Access Control Vulnerabilities . .308 CGI Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . .308 Attacking DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . .308 Information Gathering . . . . . . . . . . . . . . . . . . . . . .309 DNS Cache Poisoning . . . . . . . . . . . . . . . . . . . . . .309 DNS Cache Snooping . . . . . . . . . . . . . . . . . . . . . .310 MITM Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . .311

  xxii Contents Buffer Overflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . .313 Stack Overflows . . . . . . . . . . . . . . . . . . . . . . . . . . .314

  Heap Overflows . . . . . . . . . . . . . . . . . . . . . . . . . . .320 Integer Overflows . . . . . . . . . . . . . . . . . . . . . . . . .320 Exploiting Buffer Overflows . . . . . . . . . . . . . . . . . .321 Reverse Engineering Code . . . . . . . . . . . . . . . . . . . . .324 Executable File Formats . . . . . . . . . . . . . . . . . . . . .325 Black-Box Analysis . . . . . . . . . . . . . . . . . . . . . . . . .327 White-Box Analysis . . . . . . . . . . . . . . . . . . . . . . . .329 Application Attack Platforms . . . . . . . . . . . . . . . . . . . .332 Metasploit Exploitation Framework . . . . . . . . . . . . .333 Other Application Attack Tools . . . . . . . . . . . . . . . .336

  Defending the Application Layer . . . . . . . . . . . . . . . . . . . .336 SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .336 SSH Protocol Architecture . . . . . . . . . . . . . . . . . . .336 Common Applications of SSH . . . . . . . . . . . . . . . .338

  Pretty Good Privacy . . . . . . . . . . . . . . . . . . . . . . . . . .339 How PGP Works . . . . . . . . . . . . . . . . . . . . . . . . . .339 Key Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . .340 Securing Software . . . . . . . . . . . . . . . . . . . . . . . . . . . .340 Building Secure Software . . . . . . . . . . . . . . . . . . . .340 Security Testing Software . . . . . . . . . . . . . . . . . . . .341

  Hardening Systems . . . . . . . . . . . . . . . . . . . . . . . . . . .343 Vulnerability Scanners . . . . . . . . . . . . . . . . . . . . . . . . .346 Nessus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .346 Application-Layer Security Project: Using Nessus to Secure the Stack . . . . . . . . . . . . . . . . . . .347 Analyzing the Results . . . . . . . . . . . . . . . . . . . . . . . . .348 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .350 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .350 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .352

Chapter 9 Layer 8: The People Layer . . . . . . . . . . . . . . 353 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .354 Attacking the People Layer . . . . . . . . . . . . . . . . . . . . . . . .354 Social Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . .355

  Contents xxiii

Phone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .365

Fax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .366

Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .367

Phreaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .367

Phreak Boxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .367

Wiretapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .369

Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .369

Cell Phones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .369

  World Wide Web, E-mail, and Instant Messaging . . . . .371

Trojan Horses and Backdoors . . . . . . . . . . . . . . . . .372

Disguising Programs . . . . . . . . . . . . . . . . . . . . . . . .372

Phishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372

Domain Name Spoofing . . . . . . . . . . . . . . . . . . . . .373

Secure Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . .374

  Defending the People Layer . . . . . . . . . . . . . . . . . . . . . . . .375 Policies, Procedures, and Guidelines . . . . . . . . . . . . . . .375 Person-to-Person Authentication . . . . . . . . . . . . . . . . .377 Data Classification and Handling . . . . . . . . . . . . . . . . .377 Education,Training, and Awareness Programs . . . . . . . .378

Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .379

  

Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .381

Security Awareness Programs . . . . . . . . . . . . . . . . . .381

Evaluating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .382

Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .382

  Monitoring and Enforcement . . . . . . . . . . . . . . . . . . .383 Periodic Update of Assessment and Controls . . . . . . . . .383 Regulatory Requirements . . . . . . . . . . . . . . . . . . . . . .383

Privacy Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . .383

  

Corporate Governance Laws . . . . . . . . . . . . . . . . . .386

Making the Case for Stronger Security . . . . . . . . . . . . . . .390 Risk Management . . . . . . . . . . . . . . . . . . . . . . . . . . . .390

Asset Identification and Valuation . . . . . . . . . . . . . .390

  

Threat Assessment . . . . . . . . . . . . . . . . . . . . . . . . . .392

Impact Definition and Quantification . . . . . . . . . . .394

Control Design and Evaluation . . . . . . . . . . . . . . . .395

Residual Risk Management . . . . . . . . . . . . . . . . . .395

  xxiv Contents People Layer Security Project . . . . . . . . . . . . . . . . . . . . . .395 Orangebox Phreaking . . . . . . . . . . . . . . . . . . . . . . . . .396 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .398 Solutions Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . .398 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . .399

  Appendix A Risk Mitigation: Securing the Stack. . . . . 401 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .402 Physical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .402 Data Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .403 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .404 Transport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .405 Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .405 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .406 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .406 People . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .420 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .422

  Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423

  Foreword The first thing many people think of when they hear the word hack is some type of malicious activity. I have always thought of the term in a somewhat broader sense. Although some hacks are malicious, many others are not. Nonmalicious hacks are about exploring the details of programmable systems and learning how they really work.They are explored by those who want to understand every minute detail of a system and how to stretch the capabilities

of these systems beyond what they were originally designed to do.The nonma-

licious hacker is different from the average user or even the script kiddie who prefers to learn only the minimum necessary knowledge. Hack the Stack was written for those who seek to better understand and to gain a deeper knowl- edge of how TCP/IP systems really work. Such knowledge enables security

professionals to make systems and networks more secure and to meet the chal-

lenges that they face each day.

  

In Chapter 1, we provide you with information on how to extend OSI to

network security. In subsequent chapters, we unpeel the OSI onion layer by

layer, including a chapter on Layer 8 (the people layer).We conclude the book

with an appendix on risk mitigation.

  Let’s talk about the writing of this book. Dedicated professionals like

George Mays, Stephen Watkins, Chris Ries, Ron Bandes, and Brandon Franklin

helped make this book possible. It takes a significant amount of time to com- plete this type of task, and I am thankful to them for taking time out of their

daily work in the trenches to contribute to such an effort. After going through

this process more than once, my friends and family often ask how I have time to work, travel, and then reserve time needed to write.Well, it takes time

  xxvi Foreword

management and a desire to get it done. But as Dale Carnegie said, “If you

believe in what you are doing, then let nothing hold you up in your work.

Much of the best work of the world has been done against seeming impossibil- ities.The thing is to get the work done.”

  I hope that this book empowers you to get your own work done while facing seemingly impossible challenges.

  —Michael Gregg Chief Technology Officer Superior Solutions, Inc.

Chapter 1 Summary Solutions Fast Track Frequently Asked Questions

  Extending OSI to Network Security Solutions in this chapter: ■

  Our Approach to This Book ■

  Common Stack Attacks ■

  Mapping the OSI Model to the TCP/IP Model ■

  The Current State of IT Security ■

  Using the Information in this Book

  2 Chapter 1 • Extending OSI to Network Security Introduction “Everything old becomes new again.”The goal of this chapter is to take the well-known Open Systems Interconnect (OSI) model and use it to present security topics in a new and

unique way. While each of the subsequent chapters focuses on one individual layer, this

chapter offers a high-level overview of the entire book.

  Our Approach to This Book This book is compiled of issues and concerns that security professionals must deal with on a

daily basis. We look at common attack patterns and how they are made possible. Many