Troubleshooting Mattermost issues

Available in Mattermost Free and Starter subscription plans. Available for Mattermost Self-Hosted deployments.

This certificate summarizes mutual troubleshooting issues and techniques.

Depending on the blazon of error or trouble you lot're experiencing, refer to the sections below for troubleshooting guidance. If you're a new user, it might help to go over the installation steps once more to confirm the process.

If you lot take a paid subscription to a Mattermost offering, such equally Mattermost Professional or Mattermost Enterprise, you're entitled to open back up tickets via our online support portal.

Additionally, peer-to-peer back up is available for all Mattermost users in our troubleshooting forum and on our community server.

Of import notes

  • Do non manipulate the Mattermost database directly. Mattermost is designed to stop working if data integrity is compromised.

  • Any manipulation of the database must be done using the built in command line tools.

  • Start elementary with the step-by-pace install guides for your operating system.

Deployment troubleshooting

Docker deployments

If you're deploying the Mattermost server using Docker on an M1 Mac and encountering permission issues in the Docker container, re-create the required directories and set their permissions, and then skip the post-obit command:

                        sudo chown -R                        2000:2000 ./volumes/app/mattermost

On M1 systems, this permission change causes the deploy to stop working, and then we recommend skipping this stride birthday.

If you're experiencing issues deploying on Docker by and large, ensure the docker daemon is enabled and running:

                        sudo systemctl                        enable                        --now docker

To remove all data and settings for your Mattermost deployment:

Postgres issues

You tin can change the Postgres username and/or password (recommended) in the .env file.

Install a different version of Mattermost

  1. Close down your deployment.

  2. Run git pull to fetch any contempo changes to the repository, paying attending to any potential env.instance changes.

  3. Adjust the MATTERMOST_IMAGE_TAG in the .env file to betoken your desired enterprise or team image version.

  4. Redeploy Mattermost.

Full general troubleshooting

Some of these suggestions can exist washed directly, and others may need consultation from your network administrator.

Review Mattermost logs

You can access logs for Mattermost and use them for troubleshooting. These steps assume that you have System Admin permissions.

Mattermost Server logs

  • Ensure that log files are being created: Navigate to Organisation Console > Surround > Logging, confirm that Output logs to file is set up to true.

  • You lot can obtain the path for the log files in System Console > Environment > Logging > File Log Directory.

The resulting server log file is called mattermost.log and can exist opened with a standard text editor or shared directly.

Note

For a more complete log open Organisation Console > Environment > Logging, fix File Log Level to DEBUG, then replicate the issue to log it once again. Brand sure to revert the file log level to INFO subsequently troubleshooting to save disk space.

If filesystem access is not possible, navigate to Organization Console > Reporting > Server Logs to locate the current system logs which can be copied to a file.

You can find more on logging settings hither.

Mattermost Desktop App logs

The desktop app log file can be constitute in the user directory:

  • Windows: %userprofile%\AppData\Roaming\Mattermost\logs

  • Linux: ~/.local/share/Mattermost/logs

  • MacOS: ~/Library/Logs/Mattermost

Mattermost Browser App logs

The browser-based app does non produce additional log files. If the app has to exist debugged, use the development tools integrated in your browser for action history.

Mattermost Push Notification Service logs

Logging for the Mattermost Push Notification Service is handled via system log with logger and is appended to /var/log/syslog .

Review Mattermost environment

Put together a timeline to eliminate events prior to the error/problem occurring. For case, if you recently reconfigured your firewall and are now having connexion issues it might be worth reviewing the settings or rolling back to meet whether that resolves the problem.

  • If the problem occurred subsequent to some menstruation of normal operation, did anything alter in the surroundings?

    • Was the customer, host, or server upgraded?

    • Was an operating arrangement update applied?

    • Did the network environment alter? For example, was a server moved or a domain migrated?

    • Did the system (client or server) recently fail or abnormally terminate?

  • How many users are impacted?

    • Is this trouble affecting one, some, or all users?

    • Is the problem occurring only for a user who was recently added to the environs, such as a new employee?

    • Do differences exist between the users who are affected and the users who are not afflicted?

You tin can also search the error messages online. Existing solutions from our forum can often be found and applied.

Connect with some other device

  • If you have another mobile device bachelor, try connecting with that to see if your event still reproduces.

  • If you don't have another device bachelor, cheque with other teammates to see if they are having the aforementioned issue.

Opening a support ticket for self-hosted deployments

If y'all have a paid subscription to a Mattermost offering, such every bit Mattermost Professional or Mattermost Enterprise, y'all're entitled to open back up tickets via our online support portal.

When opening a Back up ticket as function of your paid subscription, it'due south of import that you provide us with as much information as you can in a timely manner. Knowing what data is relevant can be confusing. We utilize the anagram C.L.U.E.S. to recollect what we need:

  • Configurations

  • Logs

  • Users affected

  • Surround

  • Steps to reproduce

C.Fifty.U.Eastward.S. represents all of the data that can analyze your effect. With these details, nosotros tin can begin searching for a cause, whether it's a simple configuration change or a product issues. It likewise helps u.s.a. when we need to escalate the issue to our developers so they can spend as much time as possible improving our production.

Full general guidelines for data

Follow these guidelines when providing diagnostic data to us:

  • Make sure the files yous provide are every bit complete as possible, rather than providing a few lines. Entire log files and configurations provide us with important context.

  • Provide configuration and log files in plaintext format if possible, every bit these are far easier for u.s. to search than screenshots.

  • Be sure to sanitize configuration and log files to remove usernames, passwords, and LDAP groups. Replace these details with instance strings that contain the aforementioned special characters if possible, as special characters are common causes of configuration errors.

  • Provide screenshots or screen recordings of unexpected product behavior so that nosotros know exactly what your users are seeing.

Configuration

Why we need your configuration information

On Linux systems, settings are mostly stored in configuration files. Many issues tin exist resolved past enabling or disabling a configuration setting. In order to find a resolution, we demand to take as consummate a movie of your system setup as possible. This also helps the states to reproduce bugs so our developers can ready them.

What configuration data includes

Configuration includes (only is not limited to):

  • The Mattermost config.json file.

  • The configuration for the reverse proxy, e.g. NGINX, HAProxy, AWS.

  • The database configuration.

  • SAML configuration when the issue is regarding SAML authentication. The configuration for the Mattermost service is in the SAML IdP.

  • Whatsoever other systems that Mattermost connects to or systems that exist between the user and the Mattermost server.

How to access your configuration information

Mattermost configuration

The Mattermost configuration is ordinarily stored at /opt/mattermost/config/config.json . If you've migrated the Mattermost configuration to the database, you tin can go the configuration using mmctl or by running this database query:

                          SELECT Value FROM Configurations WHERE Active = ane;

Reverse Proxy configuration

NGINX commonly splits its configuration into 2 parts: the main server configuration at /etc/nginx/nginx.conf , and a virtual server configuration. On Ubuntu, this is stored in /etc/nginx/sites-available . Providing both of these configuration files is helpful, just providing the latter is more important.

SAML configuration

If the issue you're seeing is with SAML login, we will need to see the full configuration for the Mattermost service in the SAML provider. The configuration for the Mattermost service is in the SAML IdP. Providing screenshots similar to the ones in the setup documentation is sufficient because almost SAML providers are configured using a spider web interface.

LDAP configuration

The LDAP administrator should confirm the correct values for the following Mattermost LDAP settings:

  • LDAP server hostname.

  • LDAP connexion port, security, and certificates.

  • BaseDN, bind username, and demark password.

  • User, Grouping, Guest, and Admin filters.

  • Display attributes.

These tin exist provided as a text file or as screenshots from the LDAP server.

Other configurations

If y'all're experiencing an issue on mobile, and you're using an MDM or VPN to connect to the server, those configurations volition be necessary to diagnose the problem. A Arrangement Administrator for the external system should be able to provide you with the configuration.

Logs

Why we demand them

Virtually all computer systems take logs of errors and application behavior that can show us what's happening when an application is running. Error logs are invaluable when diagnosing a trouble, simply merely if they're every bit complete equally possible.

What logs are available

Mattermost

Mattermost has ii log files, ane for general letters and the other for notification-related messages. These are establish at:

  • /opt/mattermost/logs/mattermost.log

  • /opt/mattermost/logs/notification.log

Proxy

The location of these depend on your proxy configuration, merely a adept place to beginning looking is in /var/log . Your proxy ambassador should be able to assist you find the logs.

Database

MySQL and PostgreSQL have different logs, and their location varies based on your configuration. If the effect is related to database connectivity, cheque the database documentation to locate the logs.

SAML, LDAP, and other systems

Your organization's System Administrator should be able to find these for you.

How to access logs

Mattermost

Make sure debug logging is enabled so that we can become the most information from the logs. To do this, go to Arrangement Console > Environment > Logging, and so fix console and file log level to DEBUG.

If the behavior started at a known time or date, employ journalctl to get the logs like this:

                          sudo journalctl -u mattermost --since "2020-08-23 17:15:00" > mattermost_journalctl.log

Replace 2020-08-23 17:15:00 with the date and fourth dimension (relative to the server) when the behavior started. To become the server time, apply the date command. If the log files generated are too big to send, compress them with this command:

                          tar -czf /tmp/mattermost.log.tgz

The compressed logs will be located on the server at /tmp/mattermost.log.tgz .

If the compressed file is notwithstanding also big, use these commands to separate the compressed file into two or more than 20MB files:

                          mkdir -p /tmp/mattermost-logs cd /tmp/mattermost-logs tar czf - /opt/mattermost/logs/mattermost.log | carve up -b 20m - mattermost.log.tgz.

The compressed files will be located on the server at /tmp/mattermost-logs and be named mattermost.log.tgz.aa , mattermost.log.tgz.ab , and so on. Use a file transfer customer that supports SSH/SFTP, such as Cyberduck, to copy these files from the server.

If yous are experiencing issues with Elasticsearch, SAML, LDAP, or the database, you can enable trace logging in config.json by setting Trace to true under their respective settings. Combining this with DEBUG level file log output volition upshot in huge log files, so only go out trace logging on long enough to replicate the behavior. The resulting logs will as well incorporate a lot more than sensitive data, including user data, so be sure to sanitize it completely before sharing it with u.s..

System logs

The location of log files for other systems varies, but a practiced way to get the logs for all processes on the Mattermost server is to use journalctl like this:

                          sudo journalctl --since "2020-08-23 17:15:00" > mattermost_journalctl.log

Replace 2020-08-23 17:15:00` with the date and time (relative to the server) when the error occurred. Yous can apply --until with the aforementioned timestamp format to go the logs between two times:

                          sudo journalctl --since "2020-08-23 17:15:00" --until "2020-08-23 16:thirty:00" > mattermost_journalctl.log

Users affected

Why we demand it

Mattermost servers are chaotic places. Thousands of posts, websocket deportment, and webhook calls happen every second while users tin can be in dozens of channels across multiple teams. Knowing which users are affected by a trouble can help united states of america sift through all this information to find the root cause.

What information to include

This should be a detailed explanation of anything the finish users who are reporting the unexpected behavior accept in mutual. This includes (but is not limited to):

  • Team and Aqueduct memberships, including Direct and Group Letters.

  • Authentication methods.

  • Client operating system and app versions.

  • How users connect to the Mattermost server.

  • Any other things these users have in common such as when they joined, whether their login information recently changed, or if they are being synchronized via LDAP.

Note for Agents: This information is as well required:

  • Customer name

  • Customer contacts

  • Customer license, eastward.1000. Enterprise/Professional person

  • Customer tier

Environment

Where the Mattermost server sits in your architecture has a lot of impact on potential bug. For example, a misconfigured proxy server can forbid users from connecting even if there's nothing wrong with Mattermost.

What data to include

Because of this, having a complete picture of the servers and network that the Mattermost server operates in is cardinal to solving problems. This includes (but is not limited to):

  • Mattermost version (eastward.thou. 5.28.0, v.25.5)

  • Server OS and version (e.g. RHEL7, Ubuntu xviii.04)

  • Any orchestration/automation used like Docker or Kubernetes

  • Opposite proxy and version (e.g. NGINX 1.16)

  • Database type and version (due east.g. MySQL 5.vii, PostgreSQL 12.4)

  • SAML provider (e.g. Windows Server 2012 Active Directory, Okta, KeyCloak)

  • LDAP provider (east.g. Windows Server 2016 Active Directory, Okta, OpenLDAP)

  • The type and version of any proxies or VPNs on the network that the Mattermost server is connecting through

Be as specific as possible when describing the environs. If you are seeing errors like Connection Refused be sure to include any firewalls or filtering proxies that may be on your network, either inbound or outbound.

Examples

Mattermost server

  • External hostname: mattermost.example.com

  • Internal hostname: mattermost.lan

  • Mattermost v5.28.0

  • Zoom plugin v1.four.1

  • NGINX v1.18.0

Database server

  • Internal hostname: mysql.lan

  • MySQL v5.7

  • LDAP Provider - 192.168.ane.102

  • Internal hostname: ldap.lan

  • OpenLDAP 2.four.54 (Docker container)

Mattermost servers

  • Hostnames: mm1.local.lan, mm2.local.lan, mm3.local.lan, mm4.local.lan

Mattermost server versions

  • mm1-3: five.25.4

  • mm4: 5.21.0

Proxy server

  • External hostname: mattermost.case.com

  • Internal hostname: proxy.local.lan

  • NGINX v1.16.0

Database servers

  • Hostnames: db1.local.lan, db2.local.lan, db3.local.lan

  • Principal: db1.local.lan

  • Read-But: db2.local.lan, db3.local.lan

  • MySQL v5.six

Elasticsearch server

  • Hostname: elastic.local.lan

  • Elasticsearch vii.9 with these plugins

  • analysis-icu

Steps to reproduce

What it is

If the beliefs just happens when the user performs a specific activeness, providing detailed steps to reproduce it will assist us make sure nosotros find and prepare the correct bug. These details should be equally descriptive as possible, but aught is ameliorate than a screenshot or a screen recording of the behavior.

A brusk summary of the steps to reproduce is besides helpful. If y'all desire some examples, look at the problems tickets on some Mattermost Jira tickets.

How to provide these details

macOS

Printing CMD+SHIFT+v to open up the screen recording tool and select the region of the screen you lot want to record. To take a screenshot, printing CMD+SHIFT+four and select the region to take a screenshot. The screenshot files are placed on the desktop by default.

Windows

Press CTRL+SHIFT+S to open the snipping tool to take a screenshot. If you desire to take a screen recording you lot'll demand to install tertiary-party software such as OBS.

iOS

Take a screenshot or screen recording on iPhone.

Android

Take a screenshot or record your screen on your Android device.

Appendix

A note on mobile problems

Because the mobile app doesn't have a debug manner, diagnosing issues stemming from user data requires a proxy like Charles or mitmproxy. These volition intercept and record traffic from the customer which tin can then be replayed to reproduce issues. Contact your Customer Engineer for help setting these upwards.

SAML login problems

If the issue is with SAML login, one important piece of context is the SAML login period. This contains headers and hallmark information that can reveal issues that are easy to fix. Follow these instructions to view the SAML login menstruum if you lot are experiencing SAML authentication.

Checking keys and certificates

Key and certificate files should never exist shared, but if the error indicates a problem with the format of a key or document, then you should verify the format of the keys and certificates by running this command:

                        true cat -A /path/to/key-or.cert

The output must meet these criteria exactly to be valid:

  • Start with -----BEGIN Document-----$ .

  • All lines must finish with $ . If they end with ^M$ then catechumen them to UNIX line endings with dos2unix .

  • Terminate with -----Cease Document-----$ .