Platform Installation Guide (printable)

From LongJump Support Wiki

1. Overview

This guide provides information on installation planning, configuration, and tuning of the platform for optimum performance and security.

Intended audience:

• Installation Planners
• Installers

Learn more about the latest release: ISV Release Notes

See Also these additional resources for installation and configuration:

• installation.properties
• networking.properties
• Service Provider Settings
• Installation FAQs

2. Installation Requirements

2.1 Hardware Requirements

2.2 Software Requirements

2.3 License Requirements

• One platform license is required for each instance of the Application Server.
• To obtain licenses, see Platform License.

3. Installation Planning

3.1 Platform Hardware Stack

3.1.1 Server/Network Components

The components of the hardware stack are:

• Firewall - To provide network security
• Load Balancer - To distribute traffic across web servers
• Web Servers/Application Servers - To handle the traffic
• Database Server - To store application data

3.1.2 Recommended Configuration

This diagram shows a recommended configuration:

The key points depicted in the diagram are:

• At least two web servers are deployed behind a load balancer, to allow for failover.
• Both the load balancer and the firewall are in redundant configurations, to eliminate any single point of failure.
• The database server is replicated, using MySQL statement replication or DRBD, to another server with the same capacity as the primary database server. Switching over to the secondary database server can then be either a manual process or an automatic one, using Linux clustering/HA solutions.

3.2 Platform Software Stack

This diagram shows the server processes that make up a typical platform:

Application Server

The Application Server is a web app that runs in the Apache Tomcat container, so it can be deployed as a standalone web server. That configuration is useful for testing, experimentation, and other low-volume scenarios:

(Optional) Apache HttpD

Alternatively, the Apache httpd web server can be used to serve static content. In that case, the platform runs as an application server behind the Apache httpd web server. Those instances can be running on the same server, or on different servers, so either of these configurations is possible:

MySQL Database

MySQL is a relational database with capacities for redundancy, failover, backup, distributed access, and transactional integrity. That platform makes data look like Java objects to application developers. That makes applications easier to develop. But under the covers, the reliability and robustness of a mature relational database provides for data safety and rapid access.

Replicated Database

Database replication is a virtual necessity for any successful deployment. It safeguards data, and it allows for efficient backup and reporting from the replicated database, to minimize response time to users.

Memcached

The MEMory CACHE Daemon (memcached) is an optional, but highly recommended process that stores frequently-accessed data in memory, so it can be retrieved with maximum speed.

3.3 Platform Deployment Options

This section describes deployment options that increase scalability and reliability.

Multiple Application Servers

Instances of the platform can be deployed on different servers to improve response times.

Separate servers for Apache httpd

When the Apache httpd server is used to serve static content, the Apache httpd instances can be deployed on separate servers, while platform instances run on their own servers. Learn More: Installing and Configuring Apache for Use with the Platform

Web App Accelerators

Web Application acceleration services like those provided by Akamai can also be used to improve response times and throughput. Learn More: Web Application Accelerator.

Multiple memcached servers

The MemCache Daemon can also be run on one or more separate servers. In addition to the performance improvement, that configuration makes it somewhat easier to add application servers to scale capacity later. Learn More: Configuring memcached

Separate Database Server

Running the platform and the Database on different servers is highly recommended for a production deployment. Learn More: Configuring MySQL to Run on a Separate Server

Separate Servers for Backend Processes

The backend processes used by the application server can be (and should be) deployed to at at least one "backend server" to minimize response time in the customer-facing server they would otherwise be running on. Beyond that, heavily-used services can be deployed to servers of their own, to maximize performance. Learn More: Managing Backend Services

3.4 Prototype Deployment

The following diagram shows the kind of architecture that is typical for a production system:

The key points depicted in the diagram are:

• The platform is deployed across multiple servers.
• A Load Balancer distributes traffic across the web servers.
• Apache httpd is the front-end web server for the application server.
  Learn more: Installing and Configuring Apache for Use with the Platform
• Memcached servers reduce response time by caching data in memory.
  Learn more: Configuring memcached
• The suite of memcached servers are accessed by all application servers, backend as well as front end. (Having one memcached server for each application server is a common pattern, but is not strictly necessary.)
• The critical backend processes shown here (import, export, and scheduling, which uses quartz) are all being run from a single platform instance. But additional servers can be employed, as load demands.
  Learn more: Managing Backend Services
• Document storage (which includes pictures and image files) is managed separately from the database.
• The database is running on its own server, for added performance.
  Learn more: Configuring MySQL to Run on a Separate Server
• The primary database instance and the replication instance are each running on separate servers, both for reliability and for performance of read-intensive operations.
  Learn more: Using Replication with Different Master and Slave Storage Engines
• Requests that access and update, whether coming from a user or a backend process, go to the primary database, while read-intensive operations (backups, reports, exports) are executed on the replicated database.
  Learn more: Running Reports and Storage Checks On a Replicated Database Server.

3.5 Choosing a MySQL Replication Strategy

MySQL supports several replication formats:

• Statement-Based Replication (SBR), which replicates entire SQL statements
• Row-Based Replication (RBR), which replicates only changed rows.
• Mixed-Based Replication (MBR), which is a combination of the two.

With Statement-Based Replication, SQL statements are propagated using the standard statement-based, binary logging format. That is the default replication format in the version of MySQL that ships with the platform.

Row-based binary logging is a mechanism that logs changes in individual table rows. With row-based replication, the master writes events to the binary log that indicate how individual table rows are changed.

When mixed format is in effect, statement-based logging is used by default, but automatically switches to row-based logging when it is necessary to do so.

Learn More:

• Replication: http://dev.mysql.com/doc/refman/5.1/en/replication.html
• Replication formats: http://dev.mysql.com/doc/refman/5.1/en/replication-formats.html
• Comparison of Statement-Based and Row-Based Replication:
  http://dev.mysql.com/doc/refman/5.1/en/replication-sbr-rbr.html
• How To Set Up Database Replication In MySQL:
  http://www.howtoforge.com/mysql_database_replication

3.6 Choosing a MySQL High-Availability Option

MySQL can be configured for High Availability using the following schemes supported by MySQL:

MySQL Replication

Replication provides data safety. It also lets you run reports and backups from the secondary server, reducing the load on the primary server and improving the response time seen by users. There is some exposure to data loss, since recently-seen data may not have been replicated when a failover occurs--but on the other hand, transactions don't incur the lag time added by serialized (synchronous) replication of data.

MySQL Replication + Heartbeat

Using Linux HA capabilities, MySQL can be configured for automatic IP-address failover.

Learn More: http://dev.mysql.com/doc/refman/5.1/en/ha-heartbeat.html

DRBD + MySQL Heartbeat

While the previous two schemes do statement-based replication, Distributed Replicated Block Devices (DRBD) does synchronous replication of modified disk blocks. The advantage is consistently fast failover times. And because it is synchronous, incoming data is virtually guaranteed to be replicated, for maximum protection against data loss. The disadvantage is that traffic-handling volume may be decreased, due to the serial writes (replication must complete before the original data write can finish). This mechanism is also expensive to set up and maintain.

Learn More: http://dev.mysql.com/doc/refman/5.1/en/ha-drbd.html

MySQL Replication + DRDB + MySQL Heartbeat

Replication and DRDB are not mutually exclusive. They can be used in combination with one another.

3.6.1 Comparing the Options

The following table compares those replication schemes:

Requirements	Replication	Replication + Heartbeat	DRBD + Heartbeat	Replication + DRBD + Heartbeat
Availability
Automated IP failover	No	Yes	Yes	Yes
Automated database failover	No	No	Yes	Yes
Typical failover time	User/script-dependent	Varies	< 30 seconds	< 30 seconds
Automatic resynchronization of data	No	No	Yes	Yes
Geographic redundancy support	Yes	Yes	No	Yes
Scalability
Supports Read-intensive applications	Yes	Yes	No	Yes
Supports Write-intensive applications	No	No	Yes	Yes
Maximum number of nodes per group	One master, multiple slaves	One master, multiple slaves	One active (primary), one passive (secondary) node
Maximum number of slaves	Unlimited (reads only)	Unlimited (reads only)	One (failover only)

Note: This table is based on information obtained from http://dev.mysql.com/doc/refman/5.1/en/ha-overview.html

3.6.2 MySQL Cluster Not Supported

MySQL Cluster has functionality limitations that prevent it from working with the platform. Some of the more significant limitations include:

1. Limitation on number of objects in an instance (around 20,000)
2. Temporary tables are not supported
3. Limitations on Index size
4. Savepoint and Rollback to Savepoint are not supported
5. Replication is not supported
6. Online schema changes are not supported

4. Platform Installation

4.1 Installation Files

When you extract the installable tar file, longjump_app_server is created. This folder is typically used for the installation, but it could be renamed, so the alias "{install_dir}" is used throughout these pages.

4.1.1 Pre-Installation File Hierarchy

These files and folders are created by the extraction:

LongJump_Version

Contains the version number for the installable.

apache-tomcat-6.0.20.tar.gz

The tomcat archive file, pre-configured with the default settings for the platform. This tar file is extracted by configure.pl script and used to configure and install the platform.

configure.pl

The installation script that reads the installation.properties file, installs, and configures the platform.

installation.properties

Used to configure the installation options.

db

Contains the dump files for default databases (db1000, db892085391, and relationals) that are used to install a fresh instance of the platform.

db/patches

This file contains db SQL scripts used to upgrade database instances during platform upgrades.

scripts

This folder contains the following files:

• longjump: The application server start/stop script.
• lj-license-generator: Script used to generate Platform License requests.

4.1.2 Post-Installation File Hierarchy

When the platform is installed, these additional folders are created:

tomcat

Contains a fully-configured instance Apache Tomcat, created using the settings in the installation.properties file.

docs

Store the documents, attachments, and other files uploaded to the platform.

temp

A work folder used for temporary files.

4.2 Configuring the MySQL Server

Terminology used in this section:

MySQL Server

The database server, which has MySQL Server software installed, configured and running

MySQL Client

Any other computer which connects to the database server and has the MySQL Client software installed, configured and running

• The client connects to the database server for purposes of accessing data
• For example, a webserver may be such a client

Configuration file

my.cnf, a MySQL configuration file that may be edited and used in the MySQL client or MySQL Server installation

Configure MySQL using these values:

Storage Engine default-storage-engine = innodb Learn more: The InnoDB Storage Engine Server System Variables Configuration It has been observed that READ_UNCOMMITTED does not work well with Row-Based Replication. Use one of the following configurations: a) Either STATEMENT-BASED or MIXED replication, with the Transaction Isolation level dictated by business needs. b) ROW-BASED replication with transaction_isolation = READ-COMMITTED User Configuration Create a user account with password in MySQL to be used by the Application Server The user account should have all MySQL privileges enabled on all databases The default root account that is created during the installation of MySQL can be used sql-mode Configuration For a First-Time installation: Add the following code to the my.cnf file located in the MySQL Client: sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES" Add the following code to the my.cnf file located in the MySQL Server: sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES" MySQL Configuration for UTF-8 Unicode character set Add the following code to the my.cnf file, located in the MySQL Client: default-character-set=utf8 Add the following code to the my.cnf file, located in MySQL Server: character-set-server=utf8 collation-server=utf8_general_ci The character set defines how records can be alphanumerically ordered (or grouped, sorted, filtered, indexed). Because the list of supported languages is determined by the character set available at the server level, the UTF-8 Unicode character set configuration is required as part of MySQL configuration. See Specify character settings at MySQL configuration time for details. max_allowed_packet Configuration Add the following variable to my.cnf or my.ini max_allowed_packet = 8M. See additional information regarding this variable for details. Restart the MySQL server to implement these changes 4.3 MySQL Tuning Parameters These section provides some typical MySQL configuration values that are appropriate for a production system. These values assume: 100Mb Primary Ethernet Card 32 GB Memory 2 Processors 2 Cores per Processor Switched Ethernet Port 64-bit Red Hat Enterprise Linux ES 4 4.3.1 my.cnf / my.ini Options interactive-timeout = 7200 wait-timeout = 7200 back_log = 200 max_connections = 350 max_connect_errors = 10 table_cache = 8192 max_allowed_packet = 48M binlog_cache_size = 8M max_heap_table_size = 120M sort_buffer_size = 2M join_buffer_size = 2M thread_cache = 200 thread_concurrency = 24 query_cache_size = 500M query_cache_limit = 1M ft_min_word_len = 4 thread_stack = 256K tmp_table_size = 120M 4.3.2 InnoDB-Specific Options innodb_additional_mem_pool_size = 48M innodb_buffer_pool_size = 23G innodb_data_file_path = ibdata1:10M:autoextend innodb_file_io_threads = 4 innodb_thread_concurrency = 24 innodb_flush_log_at_trx_commit = 1 innodb_log_buffer_size = 8M odb_log_file_size = 256M innodb_log_files_in_group = 3 innodb_max_dirty_pages_pct = 90 #innodb_flush_method=O_DSYNC innodb_lock_wait_timeout = 300 4.4 Installing and Upgrading the Application Server The Application Server is provided as a compressed Linux file: longjump_app_server.tar.gz. The file must be unpacked (uncompressed). Instructions are provided for First-Time Installation and Upgrades. Considerations By default, the application server is installed to this destination directory: /~dirname/longjump_app_server, where /~dirname is defined in installation.properties as the install_dir parameter Upgrades to the application server should use the same destination directory Unpacking the .tar.gz file overwrites the longjump_app_server file in the destination directory Optionally, specify an install_dir path in the installation.properties file This destination path should be used for future upgrades 4.4.1 First-Time Installation If the platform has been previously installed on this server, follow the instructions at Upgrades. Otherwise: Review the Installable Version Release Notes. Login to a command shell Set the JAVA_HOME environment variable used by the installation/upgrade script to create configuration settings. For example: export JAVA_HOME=/opt/jdk1.6.0_nn/ Go to the directory where the install will occur. For example: cd /opt/LongJump_platform Copy the archive to that directory. Unpack it: tar xvfz longjump_app_server.tar.gz The longjump_app_server/ folder is created. Go to the installation folder: cd longjump_app_server Edit installation.properties, to specify the settings for this instance. Save the file and make a copy of it. The settings must be the same for subsequent upgrades. Run the installation script: #./configure.pl The application server is installed. Set up LONGJUMP_HOME Install the Chart Handling Libraries Start the Application Server Request a Platform License for each instance of the Application Server 4.4.2 Upgrades If this is the first time the platform is being installed on this server, follow the instructions at First-Time Installation. Otherwise, follow the process described here. Important: Before beginning the upgrade process, copy the installation options in the existing installation.properties file. The values specified during the upgrade must be consistent with those defined for the previous version. Considerations Automatic tomcat backup: The upgrade script automatically backs up the installation's tomcat subdirectory. It does not backup applications or user data. For safety, back up the entire installation directory (e.g. longjump_app_server). Prompted DB backup: During the upgrade, the platform prompts for a database backup. Installation folder: In a First-Time Installation, the application server is installed in longjump_app_server, by default: If the application server is installed in a different directory, substitute that directory name in these instructions If the install_dir parameter is specified in installation.properties, use that directory name in these instructions Quartz is enabled on every install: The Scheduler service is required for platform operations, so it is enabled on every install, whether it is a fresh install or an upgrade. In consequence, you should Disable the Service wherever it is currently running, before doing the upgrade. (If you are upgrading multiple servers, Quartz needs to be disabled on each server, in turn.) To upgrade the platform Check the Installable Version Release Notes for any version-specific preparation. Login to the server that is currently running the backend Scheduler service (Quartz). Disable the Service on that server. Login to the server to be upgraded. Install the Chart Handling Libraries Set the JAVA_HOME environment variable. For example: export JAVA_HOME=/opt/jdk1.6.0_nn/ Navigate to the directory one level above the directory that contains the application server. If the server dir is /opt/LongJump_platform/longjump_app_server, then cd /opt/LongJump_platform Copy the upgrade file (longjump_app_server.vx.x.x.tar.gz) to this directory. In a command-line interface window, enter this command: tar xvfz longjump_app_server.vx.x.x.tar.gz --overwrite The longjump_app_server/ folder is overwritten. Enter cd longjump_app_server Use a text editor to open the installation.properties file, and add the system details Save the file and make a copy of it. The settings must be the same for subsequent upgrades. Run the #./configure.pl script. Start the Application Platform Server 4.5 Platform License At installation, a 30-day platform license is issued. Before that time is up, a platform license must be requested. Requests for Platform Licenses are generally honored after one business day. Learn more: Installable Version Release Notes 4.5.1 Generate a Platform License Request One license is required for each instance of the Application Server From a Linux system, login as root It is expected that the "\JAVA_HOME" folder is part of the root path If "\JAVA_HOME" is not part of the path, enter this command: SET JAVA_HOME= Path to directory where java is installed To verify that you can execute commands, enter: #java –version Note: the pound sign (#), is the command prompt in this example Set up LONGJUMP_HOME To set the LONGJUMP_HOME environment variable: To access the license generation utility, enter this command: java –cp {install_dir}/tomcat/lib/es.jar com.es.customerLicense.LicenseRequestGenerator #java -cp {install_dir}/tomcat/lib/es.jar com.es.customerLicense.LicenseRequestGenerator __ {%%} ) ) ______/ ) _______ {_____ \/_______} ( / ( ( ( ( ( (_____ ( }{____ \_____ ( ) \______/ ____) ) / _____) / / / / \/ ------------------------------------------------------------------------------------------- LongJump Application Platform License Request Generator ------------------------------------------------------------------------------------------- Important Note: A License request should be made from the server on which license needs to be installed. ------------------------------------------------------------------------------------------- Please choose one of the following options: 1. New License Request 2. Exit # Please choose one of the following options [ Enter 1 - New License Request, 2 - Exit ]:1 Checking number of CPU(s)... Please wait Check Complete. CPU(s) found on the system ::1 System will proceed generating license request for 1 CPU(s). # Do you want to proceed. [Enter y|Y for yes, n|N for No]::y Please provide the following details for a New License Request. Note :- The specified UDP port should be free for the license server to bind to. Default UPD Port number used by License Server is 9999 # 1.) Enter the UDP port you want the license server to bind to [Enter to use default port] ::7676 Checking for UDP port 7676 availability... Please wait Check Completed.UDP Port 7676 is available # 2.)Specfiy Type of Tenancy [0 - Single Tenant, 1 - Multiple Tenant ] ::1 For Development Licenses, supported tenants cannot be more than 10. # Enter maximum number of tenants you want to support [Enter -1 for unlimited tenants]::-1 Please wait while we perform initialization and validate your request... Please Verify your License Request UDP Port for License Server  ::7676 Tenancy Type  ::Multiple Tenant Number of Tenants  ::Unlimited License requested for CPU(s)  ::1 # Do you want to proceed [Enter Y/y - yes, N/n - no] ::y Generating License Request. Please wait.. License has been stored into file called request.lic under the directory :: {install_dir}/tomct/conf/RN 4.5.1.1 Requested Parameters During the process, the following parameters are requested: Port Number Required License server runs on port 9999 by default (selected with the [Enter] button) Tenancy Type Choose from Single (0) or Multiple (1) Maximum Number of Tenants If Multiple Tenancy is selected, enter the number of tenants, or (-1) for unlimited 4.5.2 Submit the License Request To submit the license request: At the completion of this process, Platform License Request file is generated: request_companyname.lic Rename the file with your domain name in place of companyname and send the file to Support@longjump.com We will respond after one business day with the licensing information. 4.5.3 Install the Platform License Copy the license file to {install_dir}/tomcat/conf/RN Delete the file called “defaults” in that directory. Rename the license file to “defaults” Restart the Application Server The new license is installed. 4.5.4 Displaying License Information To read the license parameters that tomcat is using, open a command prompt and give the following command: java –cp {install_dir}/tomcat/lib/es.jar com.es.customerLicense.ReadLicense You can also use the REST API to query each server for the license details: You must be logged as a system administrator Use the URI https://yourhostname/networking/rest/license 4.6 Installing the Chart Handling Libraries These libraries create an image of charts generated by reports, so the charts can be viewed when the report is sent in an email. 4.6.1 Install php-cli Install php-cli from PHP version 5.2, or later. 4.6.2 Install the FCImg Library Create the fcimg/ directory: cd ${TOMCAT_HOME} mkdir fcimg/ Change the directory owner to the user who will be starting tomcat: chown {platform_admin_user} fcimg/ Learn more: Running the Platform as a Non-Root User From the installation folder, copy the template.txt file to fcimg/ cp ${platform_installation_dir}/template.txt fcimg/ Download the library: Linux (32 Bit) http://fcimg.org/release/fcimg-0.1.3-linux32bit.tar.gz Linux (64 Bit) http://fcimg.org/release/fcimg-0.1.3-linux64bit.tar.gz Extract the FCImg library into ${TOMCAT_HOME}/fcimg/ (The directory is created by the installation process. It contains a template.txt file.) Make the files executable: cd ${TOMCAT_HOME} chmod +x fcimg/bin/* 4.6.3 Install MS True Type Fonts On Windows, True Type fonts are pre-installed. For RedHat Linux: Make sure the following rpm packages are installed. (Any version should do.) rpm-build wget {a package that provides the ttmkfdir utility} For example: For Fedora Core and Red Hat Enterprise Linux 4, ttmkfdir For old redhat releases, XFree86-font-utils For mandrake-8.2, freetype-tools Install the cabextract utility. For users of Fedora Core it is available from extras. Others may want to compile it themselves from source, or download the source rpm from fedora extras and rebuild. If you don't have an rpm-based distribution, compile the tool to extract the .ttf files from the .exe files, cabextract from the source files at http://www.cabextract.org.uk/ Download the latest msttcorefonts spec file http://corefonts.sourceforge.net/msttcorefonts-2.0-1.spec If you haven't done so already, set up an rpm build environment in your home directory: a. Edit $HOME/.rpmmacros b. Add the line %_topdir %(echo $HOME)/rpmbuild c. Create the directories $HOME/rpmbuild/BUILD and $HOME/rpmbuild/RPMS/noarch Build the binary rpm: rpmbuild -bb msttcorefonts-2.0-1.spec This command downloads the fonts from a Sourcforge mirror, and repackages them so that they can be easily installed. Install the newly built rpm as the root user: rpm -ivh $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.0-1.noarch.rpm Take additional steps to finish the installation, as needed: You might need to reload the X font server. Normally this is done as a part of the installation process (by chkfontpath). However in some situations it seems like you need to reload or restart the font server manually. (On Mandrake 9.0, the last argument needs to be restart rather than reload.) /sbin/service xfs reload A bug in RedHat 8.0 makes the X server lose the connection to the font server if the font server is restarted instead of reloaded. producing assorted strange behaviors (changed fonts in newly opened applications, applications hanging). Logging out and logging in again solves the problems, or just use 'reload' instead of 'restart'.}} Verify that the installation succeeded: xlsfonts | grep ^-microsoft The list includes many microsoft and monotype font names. Restart all programs that need to be aware of the new fonts. Learn more: An easy way to install Microsoft's True Type core fonts on Linux 5. Platform Security 5.1 Managing SSL Certificates 5.1.1 Obtaining an SSL Certificate The platform provides a default self-signed certificate which is used by the Application Server. To obtain and install your own SSL Certificate, make a request to a Certificate Authority (CA). An SSL certificate authenticates a website to a web browser, part of a security protocol to manage secure data exchange. The CA will accept your Certificate Signing Request and generate a certificate which identifies your website as a secured website. To create a Certificate Signing Request (CSR) 1. Create a keystore and a private key: cd {install_dir}/tomcat/conf/RN keytool -genkey -alias tomcat -keyalg RSA -keysize 2048 -keystore {keystore_filename} 2. Create a CSR from the keystore keytool -certreq -keyalg RSA -alias tomcat -file certreq.csr -keystore {keystore_filename} 3. Submit the resulting file, certreq.csr, to the CA to obtain a certificate. (When the certificate arrives, you are ready for the next step of steps.) To Install the Certificate Obtained from the CA Once you have obtained a certificate, you need to import it into the keystore. But first, in addition to your certificate, the CA might provide a Chain/Root Certificate, which must also be imported. If you have received a chain certificate from the CA, then: 1. Copy the contents of the chain certificate into a file called chain 2. Import the chain certificate into your keystore: keytool -import -alias root -keystore {keystore_filename} -trustcacerts -file chain When the chain certificate (if any) has been imported, you are ready for the final step: 3. Import the certificate received from the CA: keytool -import -alias tomcat -keystore {keystore_filename} -trustcacerts -file {certificate_filename} 5.1.2 Replacing the Default SSL Certificate To replace the certificate: Add the new certificate to this directory: {install_dir}/tomcat/conf/RN Edit {install_dir}/tomcat/conf/server.xml file Replace the following line: keystoreFile="conf/RN/thirdParty" keystorePass="algrsa" with: keystoreFile="conf/RN/your_certficate_file_name" keystorePass="your_password_for_certificate_store" Save the file Restart the application server The Application Server will now use your certificate file for communication over https. 5.1.3 Learn More Certificate Signing Request (CSR) Generation Instructions-Tomcat, at https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR227 5.2 Controlling Port Access 5.2.1 Firewall Ports Platform Application Servers are typically deployed behind a Firewall. The firewall needs to open the ports those servers use. The default ports are: Non SSL port 80 SSL port 443 The SSL port always needs to be open. If the application is to be accessed only using https, the http port can be blocked. If the platform's Sites capability will be used to provide a public URL, then the http port needs to be open. Note: For secure communication, you should always access the platform using SSL (https://yourdomain/networking/Service). The platform provides a default self-signed certificate which is used by the Application Server. This certificate can be replaced with your own certificate, purchased from a certificate-signing authority. For more instructions on that process, see Managing SSL Certificates. 5.2.2 Changing Port Assignments To change port assignments: Editg {install_dir}/tomcat/conf//tomcat/conf/server.xml Modify the port-assignment attributes: <Connector port="80“ … /> ... <Connector port=“443“ … /> Note: If you change the http port (80), you will need to specify the port number in the URLs for the document servers recorded in the database. To see those values: Mysql> SELECT document_server,import_document_server,public_document_server FROM relationals.NETWORK_GLOBAL_PROPERTIES; Those URLs are read when an application server starts, and used as the forwarding-target for document-access requests. 5.3 Running the Platform as a Non-Root User As a good security practice, it is recommended that the platform and the various software components that it required by the platform are run as a users other than root. This section explains what you need to do. 5.3.1 memcached memcached accepts the command line argument –u {username}. With that argument, memcached assumes the identity of the specified user when running, after being launched by the root user. It is recommended that this username is set to nobody in the memcached startup command. For example: /usr/local/bin/memcached -m 250 -p 11211 -d -u nobody Here, memcached is configured to run as a daemon using 250MB of cache, listening on port 11211, and running as the user nobody. 5.3.2 MySQL A typical mysql installation has the mysql server running as the user mysql. You can verify that setting using the ps command: ps auxwww | grep mysql 5.3.3 Apache httpd server Set the User and Group directives in httpd.conf to values other than root. For example: Create a group called apache and add the user apache to that group. Set the values for User and Group in httpd.conf to apache 5.3.4 LongJump The LongJump platform is based on Apache Tomcat. It can be run as a standalone AppServer or deployed behind the Apache httpd server. When run as a standalone server, the platform listens to ports 80 and 443 for http requests. To be run as a non-root user, the platform must be deployed behind Apache, and must be listening on ports > 1023 (since only the root user can bind to low-numbered ports). Learn more: Installing and Configuring Apache for Use with the Platform After configuring the platform and Apache to work together: 1. Create the group tomcat and add the user tomcat to it: /usr/sbin/groupadd tomcat /usr/sbin/useradd -g tomcat -d /home/tomcat tomcat 2. Change the password for the user tomcat: passwd tomcat 3. Change the ownership in the folder where the platform is installed to the user tomcat.     For example, if the platform is installed in the folder /usr/local/longjump_app_server: chown -R tomcat:tomcat /usr/local/longjump_app_server 4. Restrict permission on the configuration folder to the user tomcat: chmod -R 700 /usr/local/longjump_app_server/tomcat/conf 5. If upgrading an installation in which the Documents Directory folder configured in the Service Provider Settings is     located in a folder other than the installation directory (the default location), then change the ownership of that     folder to the user tomcat. For example, if the location is /var/platform_documents: chown -R tomcat:tomcat /var/platform_documents 6. Ensure that the user tomcat has sufficient permissions on the Temp Directory If the Temp directory configured in the Service Provider Settings is /tmp (the default), ensure that everyone has read and write permissions on that folder: chmod –R 777  /tmp If upgrading an installation in which the Temp Directory configured in the Service Provider Settings is not /tmp, and is used exclusively by the platform, change the ownership of the folder to the user tomcat. For example, if the location is /var/platform_tmp: chown -R tomcat:tomcat /var/platform_temp 7. When installing LongJump, login as the user tomcat instead of user root.     That way, the ownership and permissions for the installation folders are set up properly. 8. The platform is started using the command {install_dir}/scripts/longjump start.     To allow that script to be run by the root user, modify the script {install_dir}/scripts/longjump     to become user tomcat before starting the platform.     To do that, change this line: $CATALINA_HOME/bin/startup.sh     to su - tomcat -c $CATALINA_HOME/bin/startup.sh 6. Backend Services 6.1 Managing Backend Services The LongJump Platform has several backend services. Each of them can be run on a different server, to improve performance and scalability. They can be disabled, as well, if they are not used. It is highly recommended that the customer-facing web server is not used for the backend services. Instead, they should run in one or more dedicated instances of the platform, running on one more separate servers. Important: Each of the services should enabled on only one server--not on multiple servers. 6.1.1 About the Backend Services An Application Server instance can run one or more of the following services. Enabling and disabling them determines which instance they run in--or whether they are available at all. Service Default Status Description Report Scheduler Disabled Runs scheduled reports when they're due Import Disabled Imports of data into the database Export Disabled Exports data from the database Memcached Disabled Data caching mechanism (installed separately) Marketing Disabled Handles email campaigns Bounce Disabled Handles bounced emails that result from such campaigns Quartz Enabled Time keeper for all scheduled events. 6.1.2 Configuring Backend Services at Installation Backend services can be figured at installation, using the installation.properties file. It looks like this: # Specify Java Home Directory java_home = /opt/jdk1.6.0_12 # Specify MYSQL parmaters mysql_port = 3306 mysql_username = root mysql_password = mysql123 mysql_host = localhost # Specify Services to be enabled at installation/upgrade [ 0 - disable / 1 - enable ] report_scheduler_server = 1 Kicks off scheduled reports and email. import_server = 1  Imports CSV data. export_server = 1 Exports data in CSV format. marketing_server = 1 Sends marketing emails. bounce_server = 1 Handles bounced emails. quartz_server = 1 Enables scheduling on the server. memcached_server = 1 Enables memcached, if it is installed. After changing installation.properties, run the configuration script: # ./configure.pl 6.2 Manually Configuring Backend Services During installs and upgrades, it is only necessary to enable or disable a service on a platform instance. The install/upgrade process takes care of making the connections. To change a configuration for an existing installation, changes need to be made manually, by adjusting the networking.properties file(s) and restarting the Application Server(s). Note: A change in the configuration for either memcached, quartz, or the document service requires a restart of all Application Servers. The general process is: Make the changes to the networking.properties files for all Application Servers. Restart the Application Servers. These sections describe the changes to make: Configuring Document Services Configuring the Quartz Scheduler Configuring memcached 6.2.1 Configuring Document Services Important: Changing the document service configuration requires a restart of all Application Servers. Document uploads and downloads are handled by the document service. Managed documents include: Individual documents accessed through the documents tab Attachments to records File and image fields in records The document service ensures that documents can be accessed from any application server. One of the App Servers should be designated as the document server. That server needs access to the storage device that will hold the documents. Recommendation: A reliable storage device should be used for document storage--something on the order of a SAN or a RAID enabled disk array. The URL of the document server can be configured by updating the following columns in the table relationals.NETWORK_GLOBAL_PROPERTIES: document_server - Stores documents that can be accessed only by authenticated users public_document_server – Stores public documents that can be accessed without logging into the platform import_document_server – Stores documents that are uploaded using data import Notes: The typical value for those properties is http://ip_address_of_document_server/networking Changing the document server location requires a restart of all platform instances. Recommendation: All three values should be the same. In other words, exactly one server should be responsible for all of the document services. 6.2.2 Configuring the Quartz Scheduler Important: Quartz must be enabled on at least one platform instance, or the platform won't run. If you have multiple instances, Quartz should run on only one of them, unless you follow the Quartz clustering instructions below. Changing the quartz configuration requires a restart of all Application Servers. 6.2.2.1 Enabling Quartz on a Platform Instance 1. Edit {install_dir}/tomcat/webapps/networking/WEB-INF/web.xml 2. Modify the quartz servlet mapping so that: a. The config-file parameter points to the Quartz properties file: {install_dir}/tomcat/conf/RN/quartz.properties b. The start-scheduler-on-load property is set to true, to make sure that quartz is started whenever the platform starts. Example: <servlet> <servlet-name> QuartzInitializer </servlet-name> <display-name> Quartz Initializer Servlet </display-name> <servlet-class> org.quartz.ee.servlet.QuartzInitializerServlet </servlet-class> <init-param> <param-name>config-file</param-name> <param-value>C:/apache-tomcat-6.0.20/conf/RN/quartz.properties</param-value> [ ^-- Path to the properties file ] </init-param> <init-param> <param-name>shutdown-on-unload</param-name> <param-value>true</param-value> </init-param> <init-param> <param-name>start-scheduler-on-load</param-name> <param-value>true</param-value> [ ^--Enables Quartz ] </init-param> <load-on-startup> 1 </load-on-startup> </servlet> 3. Edit {install_dir}/tomcat/conf/RN/quartz.properties 4. Add the following information: a. Database user name (an admin user) b. Password for that user Example: org.quartz.dataSource.myDS.URL = jdbc:mysql://... org.quartz.dataSource.myDS.user = rootuser [database user] org.quartz.dataSource.myDS.password = password [database user password] 6.2.2.2 Moving Quartz to Another Platform Instance Suppose you have one application server instance, and you have decided to add another to handle the traffic volume. And let's say you also decide to move quartz to the new instance, to further reduce the load on the original server. To that, you disable Quartz on the first instance, and start it on the second. Here is the procedure: 1. Open {install_dir}/tomcat/webapps/networking/WEB-INF/web.xml 2. Find the quartz servlet mapping for the first instance: <servlet> <servlet-name> QuartzInitializer </servlet-name> ... <init-param> <param-name>start-scheduler-on-load</param-name> <param-value>false</param-value> [ ^-Disables Quartz ] </init-param> ... </servlet> 3. For the second instance, follow the steps above, in Setting up Quartz on a Platform Instance 6.2.2.3 Setting up a Quartz Cluster By default, Quartz is pre-configured to run as a cluster of instances running on different servers. To run successfully, however, each server must have the settings shown here. Otherwise, a Quartz instance will not know that the others exist, and will not lock job records that are in progress. Perform this configuration on any server you later add to the cluster. 1. Open {install_dir}/tomcat/conf/RN/quartz.properties 2. Check this line: org.quartz.scheduler.instanceId = AUTO [VERIFY THIS LINE] 3. Under # Configure JobStore, check these lines: ... # Configure JobStore org.quartz.jobStore.misfireThreshold = 60000 [VERIFY THESE LINES] org.quartz.jobStore.isClustered = true org.quartz.jobStore.clusterCheckinInterval = 20000 In addition, MySQL must be configured to set the appropriate transaction-isolation level and to specify row-based replication: Edit {MYSQL_INSTALLATION}/my.cnf Use the following settings: transaction-isolation = READ-COMMITTED binlog-format = ROW 6.2.3 Configuring memcached The MEMory CACHE Daemon is a high-performance, distributed-object caching system that minimizes user response time by caching application data and other elements required by programs running on the application server. The platform uses memcached to store meta information which so that they can be obtained from the cache without accessing the database e.g. Object and field definitions, layout definitions, data policies etc. (Transactional customer data is not stored in memcached--that data is cached by the database.) Data is stored in memcached using a lazy read mechanism. When some meta information is required, the platform checks to see if it is available in memcache. If it is available, the data is retrieved. If it is not available, a database read is performed, which in turn populates the entry in memcached. When the meta information changes due configuration changes by the user, the corresponding entry is flushed from memcached. Important: The configuration for every instance must have the same memcached settings--otherwise, the caches won't be synchronized. Changing the memcached configuration therefore requires a restart of all Application Servers. 6.2.3.1 Managing memcached Servers To install memcached See http://memcached.org/ for installation instructions. To start memcached Execute the memcached script located in the /bin directory: ./memcached -d start -p 11211 -u root -m 25 where: -p is the port number (11211) -u is the user (root) -m is the amount of memory in MB (25) (Change the parameters to the values used for your installation.) To stop memcached ./memcached -d stop</tt> 6.2.3.2 Configuring Application Servers to Use memcached To enable memcached Edit networking.properties Remove the leading # character from the line #MemCachedServers=localhost:11211 To change the default port The default port is 11211. To change the port assignment, modify the value(s) for MemCachedServers in networking.properties. To configure multiple memcached instances Multiple memcached instances are recommended, so that cache access is distributed. Do that by specifying the IP address and port for each memcache server in the networking.properties file. Important: If there are multiple memcached servers, then: Every instance should have an identical list of servers. The servers should be in the same order in each list. If the configuration changes, all application servers should be restarted. To bring up the servers After making the changes to the networking.properties files for all Application Servers, (re)start servers in the required Server Restart Sequence. 7. Performance Optimizations 7.1 Configuring MySQL to Run on a Separate Server This procedure sets up database connection pooling between the two servers. Enable permissions on MySQL, so web servers can access the database server: GRANT ALL ON *.* to 'root'@'web_server_IP_Address' IDENTIFIED BY 'password' Direct the database server to the application server: Edit {install_dir}/tomcat/conf/server.xml Replace url="jdbc:mysql://{domain}/relationals" with url="jdbc:mysql://IP_address_of_database_server/relationals" Save the file If the Application server is currently running, then Restart the Application Server. Learn More: http://dev.mysql.com/doc/refman/5.1/en/replication-solutions-backups.html 7.2 Installing and Configuring Apache for Use with the Platform Apache can be installed in front of the tomcat server to serve static content like gif and jpeg images, Javascript files and stylesheets more efficiently, leaving the Application Server free to process incoming requests. This section tells you how to configure Apache to serve the platform's static content. 7.2.1 Overview With this arrangement, Apache will be in front of Tomcat. Any request that comes to the platform first goes to Apache. Apache determines if the request is a valid request and, if so, will forward the request to Tomcat for processing. Since Apache will be in front of Tomcat, it needs to be configured to listen on port 80 and port 443 for http and https traffic respectively. Apache also needs to be configured to forward traffic coming on those ports to the port on which Tomcat is listening. 7.2.2 Requirements The following Apache modules must be enabled when installing Apache. They can be compiled as a static binary into apache or loaded as dynamic modules. mod_proxy mod_proxy_connect mod_proxy_http mod_ssl mod_expires mod_headers 7.2.3 Configuration Process In outline, the process looks like this: Configure Tomcat Connectors. Configure Apache Virtual host. Start Apache, Tomcat and memcached. 7.2.4 Configure Tomcat Connectors In {install_dir}/tomcat/conf/, edit server.xml Remove connectors pointing to ports 80 and 443. Add a connector for traffic forwarded from Apache, as shown here: <Connector port="8006" maxThreads="450" minSpareThreads="25" debug="0" enableLookups="false" acceptCount="450" connectionTimeout="30000" disableUploadTimeout="true" scheme="https" proxyPort="443" useIPVhosts="true" URIEncoding="UTF-8" maxHttpHeaderSize="8192" maxPostSize="10485760" /> Notes: The platform standard is to use port 8006. That value is assumed in the remainder of this section. (It can be changed, if desired.) The next step will be to configure Apache so that incoming HTTP and HTTPS traffic on ports 80 and 443 will be forwarded to port 8006, where Tomcat will be listening. 7.2.5 Configure Apache Virtual Hosts 1. Add two virtual host entries - One in httpd.conf for Apache to listen on port 80 for HTTP traffic. - One in httpd-ssl.conf for Apache to listen on port 443 for HTTPS traffic. Any traffic meant for the platform coming on these ports is forwarded to Tomcat. This request forwarding is achieved using the mod_proxy module. The httpd.conf and httpd-ssl.conf configuration files can be found in the $APACHE_HOME directory. For each virtual host entry, make sure the following are set appropriately: ServerAdmin root@localhost DocumentRoot "/usr/local/apache/htdocs/" ServerName example.com ServerAlias www.example.com 2. Configure certificates for SSL virtual host on port 443 Make sure the following points to your certificate and private key. httpd-ssl.conf: SSLCertificateFile "{install_dir}/tomcat/conf/RN/<yourdomain>.cer" SSLCertificateKeyFile "{install_dir}/tomcat/conf/RN/<yourdomain>_private_key" 3. Enable secure communications Set up a secure communications channel between Apache and Tomcat. The URLs for the Proxy are then https://, rather than http://, and all traffic is encrypted. SSLProxyEngine on 4. Configure Proxy Paths in Apache Proxy paths need to be configured to enable mod_proxy to recognize the request and forward it to Tomcat appropriately. For example: ProxyPass /networking/* https://tomcat-host-name:8006/networking/* ProxyPassReverse /networking/* https://tomcat-host-name:8006/networking/* Note: These instructions assume that Tomcat and Apache are running on the same server. If they are running on different servers, change the settings appropriately. 5.Configure virtual hosts For each virtual host in httpd.conf and httpd-ssl.conf for ports 80 and 443, copy the settings from {install_dir}/httpd-proxy.conf, changing the standard port assignment of 8006, if needed. 7.3 Adding Additional Application Servers Adding additional instances of the platform can help to load balance traffic and improve performance. 7.3.1 Considerations Paths: It’s a good practice to use the same paths for platform installation folders and files on each server. Distribution of backend services: Running backend services on different servers can help to balance the load. For example, the Report, Bounce, and Import services can run on one server, while and the rest of the services run elsewhere. Learn More: Enabling and Disabling Backend Services Single-instance services: Each backend service should run on only one server. It is therefore important to ensure that no service is enabled on more than one server. An additional install behaves like an upgrade: When the installation process finds that the database already exists, it does an "upgrade", rather than a "fresh install". That consideration has several implications: The version of the platform installed on the new server must match the version installed on the existing servers. Otherwise, the database will be automatically upgraded, and the existing instances will find that they can no longer access the database. Any backend service settings specified in the installation.properties file are ignored. Since it's not possible to enable selected backend services during the install, they will need to be Manually Configured after the install. Note: If the service is already running on an another server, you should Disable the Backend Service first. Quartz is enabled on every install: The Scheduler service is required for platform operations, so it is enabled on every install, whether it is a fresh install or an upgrade. In consequence, you should Disable the Service on the existing server, before doing the new install. 7.3.2 Overview of the Process Disable conflicting services on the original server(s). (In particular, Quartz which will always be in "conflict".) Install the new server. Manually configure backend services it will be hosting. Manual configure changes to the original server(s), as needed, and restart them. 7.3.3 To Install an Additional Instance Disable all services on the original server(s). Download the installable file on the new server. Extract the Installable file. The longjump_app_server folder is created. Refer to the existing installation.properties file and use the same environment-variable values in the new server. (Do not use the same values for backend services.) For example: Make sure JAVA_HOME points to the JVM. Make sure mysql_host points to the database server IP-address. If there are multiple memcached servers, the list of servers should be identical, and in the same order, on all of the application servers. By default, all services are disabled in the installation.properties file, except quartz. Unless quartz will be running on the new server, make sure it is disabled. Verify database connections: Make sure the database server can be accessed from the new server. Verify the login credentials to the database server from the new instance. Run ./configure.pl to start the installation. Manually Configure Backend Services that will be running on the new server: Manually Configure Backend Services on the original instances, as needed, and bring them back up. Adding an additional memcached servers helps to distribute the cache and balance the load across the memcached servers. To add an additional memcached server Edit networking.properties on all instances. Update the MemCachedServers entry Restart all servers, following the Server Restart Sequence Note: The sequence is important! Example There is one memcached server running on serverA (192.168.0.1) on port 11211. On each instance, the networking.properties has this: MemCachedServers=192.168.0.1:11211 A second memcached server is brought up on serverB (192.168.0.2) on port 11211. The networking.properties file is edited on all instances so it looks like this: MemCachedServers=192.168.0.1:11211 192.168.0.2:11211 After following the Server Restart Sequence, all instances are using the new, distributed cache. If a 3rd memcached server is added (192.168.0.3), the networking.properties file will look like this: MemCachedServers=192.168.0.1:11211 192.168.0.2:11211 192.168.0.3:11211 Again, the Server Restart Sequence is followed to enable the new configuration. 8. Application Server Administration 8.1 Server Restart Sequence When an installation employs memcached, it is important to follow this sequence when restarting servers: Stop all application Servers Stop all memcached servers Start all memcached servers Start all application servers Considerations Stopping application servers ensures that they aren't adding entries to the cache. Stopping memcached makes sure that the cache is flushed. Those two steps can occur in either order. It is the next two for which order is critical: Restarting memcached first makes sure that a clean copy of the cache is available when the application server comes up. 8.2 Start the Application Server Login to the platform # {install_dir}/scripts/longjump start 8.3 Restart the Application Server Login to the platform # {install_dir}/scripts/longjump restart 8.4 Stop the Application Server To stop the application server, you kill the Apache Tomcat container it's running in. Login in as root Execute the command # {install_dir}/scripts/longjump stop Execute # ps aux | grep tomcat If you find a process for tomcat running, then you need to kill the process (kill tomcat_process_id) Repeat this command until the system says "No such process" 8.5 Accessing the Application Server After the application server has been started, access the platform from a web browser using the URL as follows: https://yourdomain.com/networking/Service Where yourdomain.com is the name of your domain Use this Username and Password to Login to the Platform Default username is admin@longjump.com Default password is longjump The platform will request that you change your password after you login the first time. Setup the Service Configuration with these required parameters: Configure the Service Settings and specify Service and Domain names: Parameter Description Typical Value Service Name Name of the service provider Financiocorp Services Prefix for Service Name Optional, a subdomain name Learn more: About the Prefix for Service Domain Allowed characters: a-z, A-Z, 0-9, - (alphanumeric, plus hyphen) Example: service Primary Service Domain Name of the service domain, part of the URL See Domain Name for more information mydomain.com Domain URL Read Only Automatically populated, created from Prefix + Primary Service Domain service.mydomain.com service.otherdomain.com Alternate Service Domain Alternate name of the service domain The Alternate Service Domain Name provides the ability to uniquely brand the MSP product, and replace the default ISV branding otherdomain.com 8.6 Creating a Tenant After Login, create a new tenant from a web browser using the URL as follows: https://yourdomain.com/networking/Service?t=2308 Where yourdomain.com is the name of your domain 9. Database Administration 9.1 Recognizing the Platform Databases The platform installation contains the following databases: relationals - Holds information about all tenants and a global authentication table for all users. db1000 – A database-template used to create new tenants. db892085391 – The "administration tenant" used to administer other tenants. Admins for the platform log in here. db##### - Databases for other tenants, where the number in the name is the tenant's ID. (In the platform GUI, use the Accounts tab to find a tenant's ID number.) 9.2 Database Administration Tips You should have different user accounts with different access permissions. These are typical: A root user with all permissions, accessed from a specific host only. For access from other web servers, one root user for each host. An admin user with limited privileges to do a DB dump, monitor the DB server, and/or monitor the replication server. A read only user who cannot modify the database for reports, backups, and exports. A replication user that is used only by the replication process. Always do DB dumps, reporting, and exports from the replication server, as those activities introduce locks that interfere with users. Analyze queries from the replication server. Do not login as root. Do not login as root on the production server unless executing a DML or DDL. Always use transactions when executing a DML (begin- and end-rollback). 9.3 Database Backup and Recovery MySQL can be backed up using the mysqldump command - http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html Tip: The replicated database server should be used for backups. To set up for it, see Configuring MySQL to Run on a Separate Server 9.3.1 Standard Database Backup Dump the database mysqldump –uroot –pxxx –all-databases –quick –routines –result-file=dumpfile.sql Dump the database at regular intervals, using a Linux cron job Put the mysqldump command in a shell script - say, xyz.sh Setup cron job e.g. every day at 6 pm 0 18 * * * /yourscriptlocation/xyz.sh > /somedirectory/xyz.out 2>&1 Use tar to compress the resulting dump file, to save space. Maintain daily backups for ten days or so, to reclaim the space they use. Maintain monthly backup. Save the dump of last day of the month, for example, and retain the dump file for a year or so. Store long-term backups offsite. 9.3.2 Standard Database Restore Restoring all data from a dump file Drop all the databases. Restart mysqld Start the restore process and run in background: mysql –uroot –pxxxx < dumpfile.sql > dumpfile.out 2>&1 & Restoring a single database or table from a dump file Drop the old database/table: mysql –uroot –pxxxx $database < dumpfile.sql > dumpfile.out 2>&1 & 9.3.3 Backup and Restore for a Single Tenant The items to include are: The Tenant's database Records related to this Tenant from relationals.NETWORK_DATABASES, relationals.NETWORK_CUSTOMER, and relationals.NETWORK_USER_DIRECTORY Records related to this Tenant from db892085391 Tenant's documents from documents/ and public_documents/ folders in the file system. 9.3.3.1 Back Up Tenant Data Let’s say you want to back up data for a customer (tenant) whose account number is ‘1234’. Here is the process: Dump the customer database. # mysqldump -u –p -R db1234 > db1234.sql Dump records related to this customer: # mysqldump -u -p --complete-insert relationals NETWORK_CUSTOMER \ -t -w"id=1234" > network_customer.sql # mysqldump -u -p --complete-insert relationals NETWORK_USER_DIRECTORY \ -t -w"customer_id=1234" > network_userdir.sql # mysqldump -u -p --complete-insert relationals NETWORK_DATABASES \ -t -w"id=1234" > network_databases.sql # mysqldump -u -p --complete-insert db892085391 NETWORK_ACCOUNTS -t -w"number=1234" \ > network_acc.sql # mysqldump -u -p --complete-insert db892085391 NETWORK_8c5b349851c041a28d19277a12dee8f0 \ -t -w"number=1234" > network_tenants.sql If the tenant/customer is an MSP, dump their service configuration records as well: # mysqldump -u -p --complete-insert db892085391 NETWORK_ISV_SETTINGS -t -w"id=1234" \ > network_isv_settings.sql 9.3.3.2 Restore Tenant Data Restore the data for customer #1234 from the dump files created above: Create the database for the tenant if it doesn’t exist. mysql> create database db1234; Restore the data. # mysql –uroot –p db1234 < db1234.sql Restore customer records from the dumped tables: # mysql –uroot –p relationals < network_customer.sql # mysql –uroot –p relationals < network_userdir.sql # mysql –uroot –p relationals < network_databases.sql # mysql –uroot –p db892085391 < network_acc.sql # mysql –uroot –p db892085391 < network_tenants.sql (If you haven’t made changes to the relationals and db892085391 databases, you can safely choose not to restore the records from the related tables.) Set the admin user ID to "3" (the system user) in the tenant records: UPDATE db892085391.NETWORK_ACCOUNTS SET owner_id = '3' WHERE number = '1234'; UPDATE db892085391.NETWORK_8c5b349851c041a28d19277a12dee8f0 SET owner_id = '3' WHERE number = '1234'; If the tenant/customer is an MSP, restore their service configuration records, as well: # mysql –uroot –p db892085391 < network_isv_settings.sql 9.3.4 Learn More For more detailed information on database backup and recovery in MySQL, see: http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html http://www.mydigitallife.info/2007/07/21/ how-to-backup-and-restore-export-and-import-mysql-databases-tutorial 9.4 Managing Email Credits with Vertical Response As Tenants/MSPs build and execute email Campaigns, email credits are depleted, and must be refreshed. To transfer email credits from the ISV Master Account to a Tenant/MSP sub account, see the Vertical Response website. To acquire additional ISV Master Account email credits, see the Vertical Response website. ""Learn more:"" How can I purchase email credits? How do I transfer credits using the Partner Dashboard? 10. Housekeeping 10.1 Regular Maintenance Ensures Top Performance Regular housekeeping helps to keep the platform running at peak efficiency. You'll know that housekeeping is necessary when your Platform Monitoring reveals a decreased level of performance, despite your Performance Optimizations and Platform Tuning. But regular, planned maintenance can prevent such slowdowns from occurring in the first place. As part of that process, you will be: Purging and Archiving Data on the Platform Cleaning up Temporary Files on the Server 10.2 Purging and Archiving Data on the Platform To keep storage requirements to a minimum, perform these actions at regular intervals, using the platform GUI: Audit logs are purged after a retention period which can be configured. Debug logs used for development are retained for a day. When a record is deleted, it is moved to the Recycle Bin where it is purged after 30 days. Custom purge mechanisms can be configured for required objects using Calendar based data policies where the logic for purging can be specified using Java code. The platform provides a data export mechanism which can be used for data archiving. Data can also be read using the REST API and saved elsewhere. 10.3 Cleaning Up Temporary Files on the Server Every few months, it's a good idea to remove old temporary files, to keep them from accumulating on the server. To do that: Delete old CSV files in the documents folder. For example, delete *.csv files that are more than 5 days old. Delete old files in the temp folder. For example, delete files more than 5 days old. Note: The location of the documents folder and the temp folder can be found by visiting   Settings > Service Provider Settings > Service Configuration. Delete old tomcat logs. For example, delete files more than 10 days old in {install_dir}/tomcat/logs Delete old tomcat error logs. For example, delete files more than 10 days old in {install_dir}/tomcat/logs/error Delete old apache logs. For example, delete files older than 10 days in {apache_install_dir}/apache/logs 11. Platform Monitoring 11.1 Heartbeat Check The Heartbeat Check checks to make sure that the platform is still running. It's accomplished by visiting a URL which gets processed by Tomcat. (The URL is very lightweight, does not require a login, and still gets processed by Tomcat.) https://<yourdomain>/networking/rest If Tomcat is running, you will get an http return code of 200. Note: If Tomcat runs out of memory is not responding for some other reason, you won't be able to tell by monitoring the port or the process. The Heartbeat Check is the only way to know. 11.2 Monitoring Servers Any server in the system can become a bottleneck, at some point, so it's a good idea to continually monitor the health of the critical processes that run on them. This section lists the components to monitor on each server. 11.2.1 Monitoring Application Servers tomcat availability and CPU utilization. Check threads, connection pool size, sticky sessions, KeepAliveRequests, etc. GC – Allocation and de-allocation of memory on the JVM. (Monitoring and Tuning Garbage Collection) OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics) 11.2.2 Monitoring Web Servers apache-httpd availability OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics) 11.2.3 Monitoring Database Servers mysql availability. Check caches, buffers, connections, timeouts. (Monitoring and Tuning MySQL) OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics) Note: If replication is employed, scripts can be written to check the health of the replication and report replication lag times. 11.2.4 Monitoring memcached Servers memcached availability OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics) 11.2.5 Monitoring the Network Dropped packets Socket wear and tear 11.3 Monitoring Processes These pages tell you how to monitor the critical processes on each server: Monitoring and Tuning Garbage Collection Monitoring and Tuning MySQL Monitoring OS Statistics 11.3.1 Monitoring and Tuning Garbage Collection To monitor the frequency of Garbage Collection inside the JVM: $java_home/bin/jstat -gcutil pid <intervalmillis> [root@web1]# /usr/local/java/bin/jstat -gcutil 12929 5000 S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 42.66 37.32 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 37.78 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 57.70 12.82 26.96 4845 145.414 26 1.745 147.159 Tuning parameters are set in catalina.sh under CATALINA_OPTS Learn More: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html 11.3.2 Monitoring and Tuning MySQL Running on a 64-bit machine, there should be plenty of memory. 20G of memory or more should be allocated to MySQL (highly recommended). Allocate a good amount of memory to innodb_buffer_pool_size - the combined cached for data and the index. TimeOuts interactive_timeout, wait_timeout,innodb_lock_wait_timeout. Number of file descriptors (number of files that MySQL can have open) Open_files_limit. Also: Check that you have a sufficient number of file descriptors available in the OS. To monitor the number of active threads, live threads, and open threads: show processlist or show full processlist After modifying tuning parameters in my.cnf, always restart mysql. Learn More: http://dev.mysql.com/doc/refman/5.0/en/server-parameters.html 11.3.3 Monitoring OS Statistics Before tuning any stack, whether for a web server or a database server, you should know how heavily system resources are being used. To get the information, use the following commands: top top - 15:50:31 up 309 days, 21:10, 1 user, load average: 0.06, 0.16, 0.15 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12929 root 18 0 3680m 1.2g 12m S 17.6 32.3 76:19.02 java mpstat 03:51:30 PM CPU  %user  %nice %system %iowait  %irq  %soft  %idle intr/s 03:51:30 PM all 1.79 0.00 0.29 0.43 0.01 0.06 97.42 425.80 03:51:30 PM 0 3.22 0.00 0.43 0.47 0.02 0.21 95.65 151.15 03:51:30 PM 1 1.08 0.00 0.21 0.68 0.00 0.01 98.02 103.64 03:51:30 PM 2 1.65 0.00 0.28 0.38 0.00 0.02 97.68 96.45 03:51:30 PM 3 1.24 0.00 0.23 0.19 0.00 0.01 98.33 74.56 vmstat procs -----------memory----------- --swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 22264 1110200 168460 1120660 0 0 1 39 0 0 2 0 97 0 free –m total used free shared buffers cached Mem: 3945 2862 1083 0 164 1094 -/+ buffers/cache: 1603 2342 Swap: 1027 21 1005 Learn More:' http://linux.die.net/man/1/top http://linux.die.net/man/1/mpstat http://linux.die.net/man/1/vmstat http://linux.die.net/man/1/free 11.4 Monitoring Ports Experience has shown that monitoring the processes and doing a health check tends to be a lot more useful than monitoring ports. But if you need to, here are the ports to monitor: Apache-httpd - tcp/80 and tcp/443 memcached - tcp/11211 MySQL - tcp/3306 Tomcat - The ports Tomcat is listening on are listed in tomcat/conf/server.xml 11.5 Monitoring Memcached Performance statistics can be retrieved using memcached commands. Learn More:' http://code.google.com/p/memcached/wiki/NewCommands#Statistics 11.6 Monitoring Cluster--Enabled Quartz Instances This process sets up an automatic check that sends an email alert when Quartz is down. Verify that these lines are in the quartz.properties file. (They are present in new installations, but are not automatically added during upgrades.) org.quartz.scheduler.instanceId = AUTO         # Automatically assigns instance IDs to prevent conflicts ... org.quartz.jobStore.misfireThreshold = 60000   # Prevent infinite loop if a job can't start, but keeps retrying ... org.quartz.jobStore.isClustered = true         # Turn on clustering org.quartz.jobStore.clusterCheckinInterval = 20000 Download the monitor_quartz.pl (Perl) script from the platform downloads area. Set these variable values for your installation: $mysql_username - Database user name. $mysql_password - Database password. $mysql_host - Name of server the database is running on. $config_file - Absolute path to tomcat/webapps/networking/WEB-INF/web.xml $recipients - Comma-separated list of email addresses who will receive email status messages Run the script in the background. You'll get an email alert whenever the script discovers that Quartz isn't running. How it Works The lines in the properties file cause quartz to put an entry into the relationals.QRTZ_SCHEDULER_STATE table in the database. The script checks the timestamp to verify that the last stamp was within 5 minutes. (Quartz should be running once a minute. If it hasn't run in five minutes, it is considered to be down.) 12. Platform Tuning 12.1 Tuning Tomcat The platform is a web application that runs in an Apache Tomcat container. The parameters listed in this section can be modified to optimize Tomcat's performance. 12.1.1 Connector Port Parameters Thread Allocation maxThreads, minSpareThreads, maxSpareThreads, maxkeepAliveRequests Timeouts connectionTimeOut, keepAliveTimeOut, connetionLinger 12.1.2 Connection Pooling Parameters initialSize, maxActive, maxIdle, maxWait, removeAbandoned, removeAbandonedTimeout 12.1.3 Learn More http://people.apache.org/~fhanik/http.html 12.2 Tuning Report Threads The number of report threads can be configured in the networking.properties file: report_threads=2 Assigns number of threads to be used by the report server. Generally, two threads devoted to reporting is about right. If reports aren't arriving on time, consider incrementally expanding the number of threads to see if performance improves (while keeping an overall eye on system resources, to ensure that increasing the number of threads does't incur other bottlenecks. For maximum reporting efficiency (if needed), setup a dedicated backend server for reporting. Learn More: Managing Backend Services 13. Troubleshooting 13.1 Watching Application Server Logs The log files used by the platform are tomcat/logs/relationals.log and tomcat/logs/catalina.out They contain informational messages as well as exceptions that help to troubleshoot specific problems. All messages are timestamped, with a prefix that indicates the module that recorded the message.) Examples A report scheduled to run at 2:00 is not created until much later. Examining the relationals.log for entries timestamped around that time can help to diagnose the problem. (The same applies to other scheduled events.) A java.lang.OutOfMemoryError exception in catalina.out indicates that tomcat has run out of memory. Note: In general, the Heartbeat Check is the best way to monitor the availability of the application server. 13.2 Troubleshooting Quartz If quartz stops processing jobs from the quartz queue, all scheduled jobs are listed in the NETWORK_SCHEDULE_POLICIES table. Ideally, there should be only 100 or so records/jobs scheduled in NETWORK_SCHEDULE_POLICIES. Quartz lop entries can be viewed in relationals.log on the server where quartz is enabled. The log entries will look like this, where the worker threads are the ones that pick up and process jobs in the queue: <timestamp> MasterScheduler ... <timestamp> RelationalsSchedulerWorker-1 ... <timestamp> RelationalsSchedulerWorker-2 ... ... Here are some possible ways that quartz could get stuck: There are too many jobs/records in NETWORK_SCHEDULE_POLICIES Numbers in the 1,000's may cause quartz to become bogged down. You might want to determine what caused these many jobs to be inserted in the queue. If it is safe to delete them, they can be removed from the queue. Or they could be copied to a temporary table re-inserted later, at a time when there is no load on the server. Quartz might run into an error that causes the quartz thread to die. In that case, restarting the application server will bring the quartz thread back to life. Tip: Set up a cron job that gets count of entries in the NETWORK_SCHEDULE_POLICIES. If the count hasn't changed after 20 mins or so, it usually means quartz is not picking up the jobs. (Alternatively use the REST APIs to create a task that monitors the status of that table.)