||                      LXC CONTAINERFARM - HOST(n)                      |
     Fedora srvctl 2.x       ||                                                                       |
                             ||-SERVICE-++-------------------SRV------------------------++  +-FILES-+ |
         Overview            ||         ||    +------------------------------+          ||  |/..    | |
                             ||  Bind   ||    | EXAMPLE.COM              VE 1|--+       ||  |/bin   | |
                             ||         ||    |------------------------------| 2|--+    ||  |/dev   | |
   +----------------+        ||--DNS----||    | +---------------+ +--------+ |--| N|    ||  |/etc   | |
   | CLIENT / USER 1|--+     ||         ||    | |systemd lxc    | |Mariadb | |  |--|    ||  |/home  | |
   |                | 2|--+  ||  Pound  ||    | |rsyslog sshd   | |  DBs   | |  |  |    ||  |/root  | |
   | Firefox/Chrome |  | K|  ||         ||    | |      ...      | +--------+ |  |  |    ||  |/usr   | |
   | Filezilla      |  |  |  ||--HTTP---||    | |Apache  NodeJS | |/..     | |  |  |    ||  |/var   | |
   | srvctl-client  |  |  |  ||--HTTPS--||    | |Postfix Dovecot| |/etc    | |  |  |    ||  |...    | |
   | openVPN ..etc  |  |  |  ||         ||    | |      ...      | |/home   | |  |  |    ||  |       | |
   |    SFTP | SSH  |  |  |  ||Perdition||    | |php javascript | |/mnt    | |  |  |    ||  |       | |
   +---------+------+  |  |  ||         ||    | |bash ruby perl | |/srv    | |  |  |    ||  |       | |
      +------|+--------+  |  ||--POPS---||    | |      ...      | |/root   | |  |  |    ||  |       | |
         +---||+----------+  ||--IMAPS--||    | |               | |/var    | |  |  |    ||  |       | |
             |||             ||         ||    | +---------------+ +--------+ |  |  |    ||  |       | |
             |||             || Postfix ||    +---------------------+--------+  |  |    ||  +-HOME--+ |
             |||             ||         ||       +------------------|-+---------+  |    ||  |       | |
         +---+++--+          ||--SMTP---||          +---------------|-|-+----------+    ||  |-MOUNT | |
         |        |          ||         ||                          | | | ...           ||  |       | |
         |INTERNET|===IPv4===##===NAT===##==========================+=+=+====SRV-NET====##==#-NFS   | |
         |        |          ||         ||            # libvirt 10.X.0.0/16             ||  |       | |
         +--------+          ||--SSH----||                                              ||  |-SSH   | |
                             ||         ++----------------------------------------------++  +-------+ |

## Container farm orchestration and server management tool, from D250 Laboratories 
## Istvan Kiraly 2014-2016  - LaKing@D250.hu
## For comments, bugreports, feature requests, please use https://github.com/LaKing/srvctl
## Welcome! You are at the automatically generated home page of srvctl 2.x (devel)
## Written in bash, this collection of scripts can help administrators, developers, and users of Fedora on a single or multiple systems
## to turn their Fedora OS into a virtual server farm - with LXC containers - and / or install and configure applications and services, including
## Apache, Node, Perdition, Postfix, Dovecot, Bind, sshd, fail2ban, clamav, ... 
## Wordpress, Joomla!, Etherpad-lite / Codepad, logio, phpMyAdmin, ... and so on.
## The modular design allows extending it to any application. 
## In the container-farm architecture, with configuring reverse proxies, srvctl is aware of:
## HTTP/HTTPS, SMTPS, POP3S and IMAPS - on IPv4 networks. IPv6 is under construction.
## thus, it allows you to run a web/mail/dns server, with well-isolated mini sites. 
## System requirements: Fedora 20/21 workstation or server, with fast IO on SSD's.
## Free open software, GNU GENERAL PUBLIC LICENSE.
## Active in development, engineering changes may happen, users, better said system engineers and admins are advised to message me,
## and to participate in development if using srvctl.  
## install srvctl 


## Once installed, you can use it with the following syntax


## or, in short


## for root, COMMAND is always mandatory, ARGUMENTS(s) are sometimes optional, for some commands mandatory. 
## users will triger the srvctl-client utility, that has no arguments.
## root can import bash files in a plugin-like structure, place bash scripts with srvctl command structures to /root/srvctl-includes
## srvctl help - A detailed description of ALL commands - Valid only where appropriate, some on the host(s), some on VE's

srvctl COMMAND [arguments]              


  version                               Display what srvctl version we are using.       
    Display kernel version, srvctl version
    Pound, postfix, perdition, fail2ban, bind, clamav versions

  SERVICE OP | OP SERVICE               start|stop|restart|status (enable|remove) a service via systemctl. Shortcuts for OP: +|-|!|?
    This is a shorthand syntax for frequent operations on services.
    the following are equivalent:
        systemctl status example.service
        sc example ?
    to query a service with the supershort operator "?" or with "status"
    to restart and enable a service the operator is "!" or "restart"
    to start and enable a service the operator is "+" or "start"
    to stop and disable a service the operator is "-" or "stop"

  client                                Run the srvctl client to connect to other servers.
    The client scripts allows srvctl users, or better said their clients to use a script on their workstations.
    It is Linux, Mac and Windows compatible via git bash or equivalent. It can be used to upload / download, sync files,
    and to map ports to direct ssh root access to containers. It can work in an interactive mode.

  install-mariadb                       install MariaDB (mysql) database.               
    To have an easy life with mysql / mariadb, srvctl detects an active server instance, and offers some basic, and advanced commands.
    Import or create databases, backup databases, and reset passwords. The mysql root password is stored locally, for all operations.
    PhpMyAdmin can be installed for graphical administration. 

  setup-mongodb                         MongoDB installation!                           
        Setup MongoDB repository and install latest version!

  scan                                  Run scan or phpscan and clamscan to diagnose infections even while the container is offline.
    Set of troubleshooting commands.
    Detect malicious PHP files with phpscan, files with a score above 1000 are really suspicious!
    Scan emails for viruses with clamscan, and put them innto a quarantine folder.

  diagnose                              Run a set of diagnostic commands.               
    Set of troubleshooting commands. 
    Display status messages of services, and list important network port statuses.

  exec-function COMMAND                 Execute a function in srvctl.                   
    It is possible to execute a command or a srvctl-function after bootstraping srvctl. 
    Thus, srvctl configs, constants, variables, and funcions are available.

  exec-on-hosts COMMAND                 Execute a command on every srvctl host accessible.
    Since several srvctl hosts may be available, this command will execute it on all hosts.
    The command is executed on the host itself, and on all remote hosts.

  add-dyndns HOSTNAME                   Add new domain as dyndns entry.                 
    This will create a new DNS entry that can be updated with a simple http call to the host.
    Simply send a POST request with the key returned by this command, and the hostname as URI to update the dyndns record. 

  add-fedora VE [CMS]                   Add new LXC OS-container. Optionally specify DNS
  add-codepad VE                        Add new LXC OS-container running codepad.       
  add-ubuntu VE                         Add new LXC ubuntu based OS container.          
  add-apache VE                         Add new LXC application container running apache with a readonly filesystem.
    This will add a new LXC OS container, also called virtual enviroment or VE - to a srvctl host. Each container is unique, and runs a complete OS.
    The name of the VE has to be a domain name, and might be a .local domain or a subdomain. Developer domains can be prefixed with dev. 
    An optional username can be given to define the owner of the VE. Multiple users can have access to the VE, defined in the containers users file.
    Each container will be configured with SSH keypairs, all authorized users can have root-access to a VE from the user account of the srvctl host.
    Logged in users can access files of the containers with ssh - usually in a two step hop, or directly with NFS folders mounted to their home/VE directories.
    The srvctl-client script can be used to sync, backup, upload files. Proper SSH port forwarding allows SFTP access directly from remote user computers.
    OS Containers will be configured as web and mail servers. The srvctl command will be available on every VE, and can be used to configure further.
    Local containers might be called *.local *-local or *.devel *-devel.
    Prefixes make sense, mail. or dev. will create MX or development servers.

  backup-hosts [?]                      Backup srvctl hosts / query backup status       
    Attempt to backup all container-data, but not the container operating system.
    This will create a folder - path specified in config - and rsync files and folders.
    Optional directives may be specified in /etc/srvctl/backup-hosts-include
    - local_backup DIRs
    - server_backup HOST DIRs
    - remote_backup PROXY HOST DIRs

  backup [VE]                           Backup VE data, or all containers locally.      
    Attempt to backup all user-data, but not the container operating system.
    This will create a folder - path specified in config - and create 7zip archives.
    Archives should be accessible for all authorized users.

  restore VE                            Restore VE based on backup data                 
    Attempt to restore all user-data, and create a new container operating system.

  exec VE [CMD ..] | VE [CMD ..]        Enter the root shell, or execute a command on a given container. This is the default command.
    Users can access local containers directly. Syntax is similar to that of ssh. eg.: 
        sc example.com
        sc example.com "do-something with arguments"

  exec-all 'CMD [..]'                   Execute a command on all running containers of this host.
    It is possible to iterate trough all running containers, and run a command.
    The command will be executed via ssh, as root, starting in the /root folder.
    Enclose the command into a string to use certain operators, like &&

  show-csr VE                           Show the certificate signing request, (CSR) for secure https connections of the VE.
    The certificate signing request to be used for signing. mandatory for HTTPS connections.

  import-crt [CRT]                      Import a signed certificate for secure https connections of the VE.
    The HTTPS protocol requires certificates to work. By default srvctl generates self-signed certificates.
    Signed certificates can be imported with this command. Encryped certificates are not supported.
    The CRT may contain the CA-bundle, eg. the certificate chain, the key, and the signed certificate in PEM format.

  import-ca CRT                         Import a pem format root-certificate file issued from a trusted certificate authority.
    Import CA root certificates to be used system-wide. ...

  kill VE                               Force a container to stop.                      
    This command executes lxc-stop -k, to halt a container. It is recommended to use a stop command before killing it.
    Rather than requesting a clean shutdown of the container, explicitly kill all tasks in the container.

  kill-all                              Force all containers to stop.                   
    This command executes lxc-stop -k, on all running containeres! It is recommended to use a stop / stop-all command before.
    Rather than requesting a clean shutdown of the containers, it will explicitly kill all tasks in all the containers.

  reboot VE                             Restart a container.                            
    This command will reboot the VE, by executing reboot in the container - via ssh.
    If ther container is not accessible over ssh, it can be manually stopped with lxc-stop or killed with srvctl.
    Rebooting a container with srvctl will remount NFS shares.   

  reboot-all                            Restart all containers.                         
    This command will reboot all VEs, by executing reboot in all the containers one by one, via ssh.
    If ther container is not accessible over ssh, it can be manually stopped with lxc-stop or killed with srvctl.
    Rebooting containers with srvctl will remount NFS shares.   

  regenerate [all]                      Regenerate configuration files, and restart affected services. (!)
    In case some srvctl configuration files, or data files are changed, it is required to regenerate runtime configurations.
    Configuration files are mostly located in /root/srvctl-users and /srv/VE/ but data may reside in users home folders.
    For all critical configuration files .bak backup files will be created. Regenerated important system configuration files include:
    /etc/hosts, /etc/relydomains, etc, ...
    NFS shares are mounted if missing. Note, direct mount-shares in /mnt are presistent, and reside in the users /home folders.
    Important srvctl VE configuration files located in the /srv/VE/ folder, used as base-data in the regeneration process include:
        aliases - newline seperated list of domain names that are considered alternate domain names for the container.
        users - newline seperated list of local usernames granted root access to the container. Nonexistent users will be created.
        pound-host - the primary hostname for the host used in the web-hosting. Pound is the reverse proxy used in srvctl.
        pound-http-port, pound-https-port - defines what ports of the VE should be served on the http / https port of the host.
        pound-http-service-directives, pound-https-service-directives - additional pound BackEnd configuration directives
        pound-enable-altdomain - A container may have one alternate domain name defined in that file, serving a different content.
        pound-altdomain-http-port, pound-altdomain-https-port - for the alternative domain, custom ports should be defined.
        pound-altdomain-http-service-directives, pound-altdomain-https-service-directives - pound BackEnd configuration directives.
        pound-enable-dev - if file is present, codepad VE configuration is set for pound. Port 9001 is forwarded on the dev. subdomain.
        pound-enable-log - if file is present, logio VE configuration is set for pound. Port 9003 is forwarded on the log. subdomain.
        pound-no-http - if file is present, all http requests are redirected to https by pound
        pound-no-https - if file is present, all https requests are redirected to http by pound
        pound-redirect - redirect all http and https requests to the domain name entry in that file 
        disabed - if that file is present, the container is considered diabled and wont be started by srvctl.
    Additionally, adding signed certificates to /srv/VE/cert requires a regeneration too. Certificates are checked for validity. 

  remove VE                             Remove a container.                             
    This will stop the VE and remove its files to /temp - or the temporary directory defined in /etc/srvctl/config
    Containers will be extended with the date of removal. srvctl does not delete any files.

  restart-services                      Restart all srvctl-managed services.            
    Srvctl keeps track of important services of the system. 
    This command restarts all of them, and displays if they are active, enabled, or not.

  settings VE OPTION [disable]          Set custom directives for a VE, or return to the defaults
    Set a custom directive for a VE. OPTIONS are:
    no-http - redirect all http traffic to https
    no-https - redirect all https traffic to http
    http-port NUMBER - serve http from a custom port
    https-port NUMBER - serve https from a custom port
    redirect URL - redirect incoming traffic to DOMAIN

  start VE                              Start a container.                              
    Start the VE. If the container was disabled, it will be enabled.
    The NFS share will be mounted on the hosts home folders for the configured users.

  start-all [delay]                     Start all containers and services. Optionally with a delay in-between.
    Start all, except the disabled containers. It will also mount NFS shares.
    This operation is relative CPU-intensive, and depending on the number of containers it may take several minutes.
    The delay parameter can be used to slow-down the process. Default is 1 sec.

  status                                Report status of containers.                    
    A quick query on all enabled containers.
    It will display the ping-time for each running container.

  status-all                            Detailed container status report.               
    A detailed, thus slower query of all containers, displaying fields in the following order:
        STATUS - the ping time, or none for disabled containers.
        HOSTNAME - the container name.
        IP-LOCAL - the internal IPv4 address of the container.
        IN - nslookup check for the IN A DNS record. In case of no OK the container can be reached trough ve.host-hostname.
        HTTP - query pound status information.
        RES - http-reponse status code, when querying the container. It should be 200.
        MX - nslookup ncheck for the IN MX DNS record. in case of OK mail is intercepted and forwarded to the container.
        DISK - summarizes total diskspace usage by the container.
        USERs - list of usernames granted root access to the container.

  usage                                 Container usage status report.                  
    Usage status of containers.
        HOSTNAME - the container name.
        DISK - summarizes total diskspace usage by the container.
        LOGs - Log size gives a good approximation for network traffic.  

  list                                  List containers and their internal IP information.
    A Quick list of accessible containers.

  ls                                    List containers in ls format                    
    A Quick list of acessible container names to be processed further in other scripts..

  info                                  Detailed container information                  
    A Quick list of acessible container names to be processed further in other scripts..

  stop VE                               Stop a container.                               
    Stop a container via shh and shutdown. It will also unmount NFS shares.

  disable VE                            Stop and disable container.                     
    Stop a container via shh and shutdown. It will also unmount NFS shares.
    Disabling it will prevent it from starting it with the start-all command.

  stop-all                              Stop all containers.                            
    Stop all containers via shh and shutdown. It will also unmount NFS shares.

  update-install [all]                  Update the container.                           
    This command will run the srvctl installation scripts, thus inicailize the host as a container-farm.
    With the [all] option set, all srvctl-related existing configurations will be regenerated, and updated. 
    In the first step, a blank configuration fill will be written to /etc/srvctl/config
    Following files are honored - if found:
         /root/crt.pem, /root/key.pem, /root/ca-bundle.pem - certificates for the host
         /root/saslauthd - a custom binary, that fixes the incompatibility between perdition and saslauthd
    A company domain name should be set in the config file, and a logo.png and a favicon.ico should be at that domain.
    Custom files for pound will reside in /var/www/html, and they might be customized.      

  create-certificate DOMAIN             Create an SSL certificate.                      
Generate key, make a csr, self sign it. It can be signed by a certificate authority as well.
Certificates will reside in /etc/srvctl/cert/DOMAIN 
    Note for signing SAN wildcard certificates with a CA. Srvctl is mapping / can map container ports to domain names.
    It is possible to have wildcard certificates for *.service.domain.net and *.label.domain.net
    Since srvctl maps certain applications default ports to subdomain names it is recommended to keep the following list of active and possibly future services and labels.
    Labels:    Services:
        www    test
        web    stage
        dev    codepad
        run    play
        log    logio
        ssh    shell
        sys    cockpit
        dns    zone
        git    repo
        src    source
        srv    service
        lab    label
        doc    docs
        dyn    dyndns
        ftp    files
        adm    admin
        pma    phpmyadmin
        alt    port
        opt    custom
        vnc    container
        vpn    network
        gui    devel
        wss    websocket
    Additional labels:
        .. feel free to submit suggestions on github.

  add-publickey [keyfile]               Add an ssh-rsa public key.                      
    Public keys allow passwordless ssh access. Srvctl manages key and access to servers and containers.

  add-user USERNAME [VE]                Add a new user to the system. Optionally, grant the user access to VE.
    This command will set up a new user account. It will generate a password, and password hashes for VE applications.
    A single user may have access to all containers and their CMSs, with the same password. The generated password is stored in plaintext.

  add-cms [CMS]                         Install a content managment system.             
    Use the github release of Joomla! Create configuration files.

    Use ShellChek and beautiy-bash to syntax check your bash project tree and to format.

  install-node                          update install the latest node.js               
    Goes to nodesource, and checks updates installs the latest version.

  regenerate                            Restore permissions on important files and folders.
    Set ownership, and mode on files and folders, as good as possible.

  setup-codepad [apache|node]           Install etherpad and codepad and start a new project. The command setup-codepad-release will use the latest etherpad release instead of git.
    Install, and set up codepad a collaborative code editor, or better said a collaborative online development environment.
    It should be reached on the dev. subdomain with https - this however needs to enabled on the host. (pound-enable-dev)
    Default is to create a node project, and a basic hello world application. 
    Homepage: http://codepad.etherpad.org/ and http://D250.hu

  setup-logio                           Install log.io, a web-browser based realtime log monitoring tool.
    Install, and set up log.io to access logs from a browser.
    It should be reached on the log. subdomain with http - this however needs to enabled on the host. (pound-enable-log)
    Homepage: http://logio.org/


  joomla [path]                         Install the latest Joomla! Optionally to a folder (URI).
    Use the github release of Joomla! Create configuration files.

  nodebb                                Install NodeBB                                  
    Use the github release of Joomla! Create configuration files.

  wordpress [path]                      Install Wordpress. Optionally to a folder (URI).
    Install the latest wordpress from wordpress.org, and create configuration files.
    Homepage: https://wordpress.org/

Fedora srvctl by Istvan Kiraly - LaKing@D250.hu - D250 Laboratories - 2015