Principal Infrastructure Engineer - HPE, 2022-present

• Team Lead - AI Solutions
• Formed Infrastructure team for new organization
• Establish new team procedures
• Establish relationships and build trust across multiple orgs
• Develop interview questions and process for new team
• Coordinate technical merge of two acquired Startups into HPE enterprise processes
• Manage multiple external vendor relationships
• Keep up with work from multiple teammates and work being done by other areas to inform planning and anticipate potential conflicts
• Team managed CircleCI, Github Actions, Dockerhub, Gemfury, Pypi releases; GCP, Azure, AWS clouds; bare metal; user provisioning; etc
• Implement new AWS org structure from AWS contract to account definition for multiple products
• Merge multiple GCP orgs into single centrally-billed org with new project structure
• Build new Azure Tenant structure and migrate existing plans
• Implement automated onboarding procedures for systems with local user management (Ansible, gitops, Jira, Slackbot)
• Imlement self-service Gitops Terraform (using Github Actions, Digger and Jira) for most service management, including AWS services, Github repos, Dockerhub access, DNS zones and entries, GKE-based k8sc clusters, etc
• Implement centralized federated role-based authentication using Azure Entra ID to authenticate Github Enterprise, AWS via Identicy Center, Teleport for machine and k8s access, etc
• Build new Github Enterprise account to contain auth for multiple separate Github organizations
• Respond to internal security incidents and security audits for cloud systems, including external SaaS and developer systems
• Implement self-hosted Renovate to automate patching, augmented by Dependabot and Snyk for Github security scanning
• Implement uniform cost control / tagging standard across clouds, including cost management tools (CloudChipr and cloud-specific)
• All sorts of other things; we operated like a separate startup in a lot of ways, and part of HPE in others
• Automate Everything :)

Staff Site Reliability Engineer - Kong, 2020-2022

• Manage AWS / EKS-based infrastructure supporting SaaS product
• Implement new download site
• Dockerize and open source individual Pulp containers
• Work across product teams to gather requirements / build consensus
• Develop automated testing and deployment process with Argo, Github Actions, Robot
• Presented infra design and lessons learned at PulpCon 2021
• Act as SRE SPOC for Security and Compliance teams (including primary responder to compliance audits; FedRAMP/SOC2/Pen tests)
• Work cross-team to gather requirements and lead work efforts
• Open issues and contribute to upstream projects
• Create on-boarding documentation and drive formalization of procedures (document decisions with ADRs, use change management, mandatory github code review, etc)
• Mentor more junior teammates
• Periodically present training and team work updates to varied audiences
• Participate in interview pool for new candidates
• Automate Everything :)

Senior Software Engineer - SUSE, 2018-2020

• Write code and solve problems in the SUSE CaaS Platform (CaaSP)
  • Typical technologies: Kubernetes, Ruby / Rails. Python, Jenkins, etcd, Salt, SUSE Linux
  • Implement feature requests like OIDC configuration and SSL cert management, bugfixes
• Open issues and contribute fixes to OpenSUSE and upstream Open Source projects
• Provide "sysadmin" perspective within developer team
• Mentor more junior teammates
• Conduct training sessions / demos on new solutions
• Provide engineering support to L3 engineers supporting customers
• Employ multiple communication mechanisms to participate in Agile development process with world-wide dispersed team
• Act as team lead for "bug squad" and "feature squad"
• Act as North America Release Engineer
  • Coordinate release activities with developers in America timezones and release engineers in Europe & Asia
  • Package software as RPM and containers for SUSE releases using Open Build Service, Kiwi, Jenkins, and GitHub Actions
  • Maintain CI pipeline (Jenkins) code
  • Coordinate security / bugfix releases of customer-facing code
• Act as CaaSP Security Champion
  • Pioneer and advocate Security Champion role within SUSE
  • Serve as single Point Of Contact for Security within CaaSP team, and as SPOC for CaaSP issues within Security team
  • Prioritize security bugfixes like CVEs within team
  • Champion security-related work, such as improving use of capabilities within control plane containers and implementing automated execution of scanners like kube-hunter in CI process

Senior Linux OS / Automation Engineer - State Farm Insurance, 2015-2018

• Convert existing CFEngine 3 infrastructure to Puppet 3 (then 5)
• Convert several existing one-time configuration scripts to continuous-validation native Puppet logic (occasionally using Ruby facts/functions and defined types)
• Develop and tune PostgreSQL configuration management database
• Develop utilities and scalable, geographically-distributed web services using ksh93 (bash), perl, and python languages, backed by etcd and PostgreSQL data store
• Provide third-level enterprise Linux OS support (>200K virtual machines plus a couple thousand physical machines; mix of RHEL/CentOS and Ubuntu)
• Lead Linux OS component of hybrid public cloud migration
• Build / automate base Linux OS images (Ubuntu and Amazon Linux)
• Provide direction on patch management, security scanning, connectivity, and lifecycle for EC2 instances
• Implement enterprise-wide Prometheus / Grafana / ElasticSearch centralized monitoring / logging / alerting system
• Research trends and investigate new solutions continuously
• Provide consultation on various Linux OS configuration and automation issues to business partners through project assignments and ad-hoc engagement
• Employ Kanban and SAFE Agile methodologies for service-oriented work
• Work within a geographically-distributed team which unified formerly separate UNIX/Linux Security and OS support
• Conduct training for team and second-level support / mentor new team members
• Plan/participate in migration to new compute environment (host naming convention, system bootstrap design, secure provisioning, etc)
• Participate in third-level 24x7 on-call rotation
• Respond to incidents escalated from second-level support
• Own and resolve problem records
• Develop knowledge items for use by first- and second-level teams
• Participate in disaster recovery exercises
• troubleshoot and recover physical and virtual machines from bare metal
• use PXE boot, file backups, and several custom scripts in isolated environment
• Use ITIL-based change control process to deploy environmental changes
• Automate software deployments using in-house Gitlab/Jenkens infrastructure

Tier 3 Unix Security Problem Management - State Farm Insurance, 2006-2015
(contract via TekSystems 2006-2008; direct 2008-2015)

• architect and oversee deployment of CFEngine to a mixed Unix environment
• Planned multi-year project to convert ~80K lines of shell and perl scripts to native CFEngine policy / self-service web application
• Developed database back end to serve as central store for security configuration and all security configuration information (sudo rules, account attributes, system access permissions, system classification, ssh keys, etc)
• Designed and implemented software version control repository for all CFEngine and Webmin module code, with hooks for automated deployment, defect tracking integration, and peer review, and transparent structured deployment
• Designed and implemented self-service web applications using Webmin framework and several custom modules (Perl) for ~50K end-users
• Designed and implemented active-active management cluster with geographic fail-over, primarily using Gluster and Postgres behind pgpool-II with etcd-based heartbeat
• Replaced legacy form-based non-automated system
• Added complete accountability for all security attributes on all managed Unix platforms, improving auditability
• Enabled systems to self-repair all known security configuration items (~43,000 checks done every 15 minutes as of mid-2013, will be over 100K by project completion)
• Integrated with concurrently-developed Puppet environment.
• Install CFEngine through Puppet manifest
• Develop custom facts using Ruby and JSON to share information between CFEngine and Puppet
• Work with Puppet support area through regular sync meetings to ensure appropriate separation of duties and avoid managed resource collisions
• diagnose and resolve unique issues on Unix platform (HP-UX, AIX, Linux (primarily RHEL and SLE))
• share knowledge of advanced Bourne shell and Perl scripting
• develop scripting style guide and code beautifier for ksh scripts
• develop scripts for first and second level tech support staff to use, primarily in Ksh and Perl
• developed multi-platform single-pass user synchronization tool (soon to be open-sourced)
• completely rewrote interactive user management tool
• developed automated system security auditing tool
• developed several tools to enforce consistency of security configuration, such as sudo management, kerberos configuration, password generation, ssh key management, etc
• Participate in third-level 24x7 on-call rotation
• Respond to incidents escalated from second-level support
• Respond to incidents directly opened by business partners and automation
• Own and resolve problem records as problem management team
• Develop knowledge items and procedures for use by first- and second-level teams
• Own and maintain configuration items
• Deploy environmental changes following ITIL-based change control process
• mentor newly-hired / less experienced security analysts
• advocate and drive adoption of open source software products
• work with business partners with varying backgrounds to identify and communicate potentially complicated technical and security issues
• provide Unix security consultation for business partners
• maintain high-level knowledge of various regulatory / audit requirements inherent in a domestic multi-line insurance and banking company which processes credit card payments (PCI, SOX, GLBA, HIPAA, etc)
• set policy for future Unix environment growth and usage, accounting for management tool scalability and anticipated business requirements

Technical Reviewer - Packt Publishing, 2014

• Identify and correct technical issues in PostgreSQL Cookbook​ (ISBN 139781783555338)
• Provide feedback on draft copies of chapters within 2-3 days of receiving the chapters
• Topics included common Postgres administration tasks, including data optimization, interaction using various programming languages, and management of extensions.

Technical Reviewer - Packt Publishing, 2013

• Identify technical issues in Webmin Administrator's Cookbook (ISBN 9781849515849)
• Provide feedback on draft copies of chapters within 2-3 days of receiving the chapters
• Topics were common UNIX (Primarily Linux) cross-platform administration tasks, often performed through a web interface
• Several pieces of feedback were included as sidebars in the final book

Applications Benchmarking Engineer - Intel Corp., 2005-2006
(contract position)

• Assemble, configure, test and debug production and pre-production hardware in a clustered, high-performance environment, generally running RHEL 3.x and 4.x as well as SUSE Linux
• Configure HP and Linksys managed switches for performance Gigabit Ethernet interconnects, as well as managing and configuring Myrinet and Infiniband interconnects using a variety of products
• Develop new and maintain existing scripts for automation and validation purposes, using primarily Perl and Bash.
• Rewrite, reorganize, and expand roughly 12,000 lines of Perl, finishing and correcting the rough implementation started by a previous contractor while making the coding style consistent and more modular. 90% of the script was rewritten, and the codebase roughly doubled in size from the new features.
• develop moderately complex Perl framework for gathering, collating, and graphing performance benchmarking information
• Actively share knowledge of Linux operating system and Perl with members of the Cluster Development team as well as with the local IT department.
• Assist in diagnostics of other computing areas as appropriate

Network Systems Administrator - Teleologic Learning Co., 2000-2005

• Maintain all Internet services, including www, smtp, ftp, cvs, subversion, mysql, DNS, etc. Manage redundant servers in most cases.
• Research, recommend, and assemble new network workstations
• Maintain mixed Win9x/Mac/Linux network workstations
• Provide workstation support for local and remote (either at home offices or connected via VPN) employees
• Developed a ~2000 line Perl script and an associated markup language which reduced roughly three weeks worth of work down to under one minute, and reduced web developer staffing requirements 30% while greatly increasing Content Developer productivity and content reusability
• Developed a tree-based Java program distributed via Java WebStart which allowed simple editing of fields within a structured XML file from within a .zip file without requiring the end user to know that they were using XML or .zip (SCORM-compliant manifests edited from within packages SCOs)
• Maintain and upgrade internal data network and phone system, including basic configuration of managed switches and Cisco equipment
• Maintain Internet connectivity, including the planning and construction of Linux based firewalls (both whitebox and embedded) using iptables and ipchains rules generated by hand.
• Automate server and network device tasks using scripts written in a variety of scripting languages, generally Bourne shell and Perl.
• Monitor system and network uptime using custom scripts as well as pre-written systems such as MRTG, NetSaint/Nagios, Big Brother, NetSNMP, etc.
• Manage security patches on various Linux distributions, Windows, and OS X, as well as performing periodic security audits / penetration testing using tools such as nmap, and Saint.
• Monitor network security using Intrusion Detection tools including Snort and Acid, as well as diagnostic tools such as Ethereal, Etherape, ntop, etc
• Develop server-side and client-side web scripts using PHP, JavaScript, Perl, MySQL, HTML, Cold Fusion
• Provide JavaScript and PHP training to developers

Unix Administrator - Parkland College, 1999-2000

• Plan and implement conversion of student BSDi labs over to SuSE Linux
• Research and implement web-based ~10K student email system using LDAP, Courier IMAP, Postfix, and Gnu Horde
• Develop custom LDAP user management utilities using Perl and Net::LDAP both for CLI and web-based tools, some of which were open-sourced and given back to the Internet community
• Maintain Linux lab and student email system
• Provide basic faculty Linux training
• Supervise and train student Linux administrator

Part-time Microcomputer Specialist - Parkland College, 1998-1999

• Install + maintain Win 9x lab and faculty workstations
• Provide application help-desk services
• Co-supervise student employees

Student Microcomputer Specialist - Parkland College, 1998

• Assist with maintenance of Win 9x lab and desktop workstations

Stack Exchange network profile

https://stackexchange.com/users/25511/dannysauer

LinkedIn

https://linkedin.com/in/dannysauer/

GitHub

https://github.com/dannysauer

Operating Systems

• Linux - 29+ years

  • LPIC-1 (Linux Professional Institute Certified)
  • LPIC-2 (Linux Professional Institute Certified)
  • SuSE, Slackware, YellowDog, LinuxPPC, Redhat (RHEL, Redhat, and Fedora), Gentoo, Ubuntu, Debian, Knoppix, custom systems, etc
• HP-UX - 10 years
• AIX - 10 years

Programming Languages / Domain-Specific Languages

• Perl, Bourne shell (ksh, bash, posix sh), CFEngine, Puppet, Terraform, awk, PHP, SQL (Postgres, MySQL), HTML, JavaScript, Puppet, Java, C, C++, PIC assembler, Python, Ruby

Web Servers

• Apache, nginx, Boa, Thttpd, Roxen, gunicorn, haproxy, who even reads this section?

Email Systems

• Postfix, Sendmail, Procmail, Courier Imap, Qpopper, Imp, IMHO, SquirrelMail, RoundCube, SpamAssassin, DSpam, DBMail

Various

• navigation of large organizations, working independently and with a team, working to help people with varying levels of familiarity to understand complex technical concepts, writing really bad resumes
• network debugging, intrusion detection, various Internet services, backup/recovery, PC component assembly, network design, DHCP, DNS, LDAP, NIS, Samba, rsync, PIC mid-range microcontrollers, Arduino, HPC, RAID, LVM, EVMS, PPP. Some Cisco, including PIX firewalls, Unifi

University of Illinois at Springfield

• BS Computer Science, security emphasis