link管理
链接快照平台
- 输入网页链接，自动生成快照
- 标签化管理网页链接
相关文章推荐
|
慷慨的铁板烧 · js获取Android和iOS的版本 | ...· 3 周前 · |
|
|
空虚的显示器 · Flask 从入门到放弃4: Web表单 ...· 2 周前 · |
|
|
爱热闹的葫芦 · Blog Posts on nuxt · ...· 2 周前 · |
|
|
阳光的青蛙 · 实现程序自启动的最佳实践 | ...· 1 周前 · |
|
|
知识渊博的蟠桃 · 记一次Font导致JVM堆外内存泄漏分析 ...· 1 周前 · |
|
|
暗恋学妹的吐司 · 烂柯奇缘txt下载笔趣阁_烂柯棋缘(真费事) ...· 4 月前 · |
|
|
乖乖的皮带 · jsk工房36款小游戏-jsk工房36款小游 ...· 7 月前 · |
|
|
奔跑的啄木鸟 · 揭秘：俄军信息化联合一体作战“大脑” ...· 8 月前 · |
|
|
才高八斗的豆浆 · 防爆伺服电机|气相色谱仪-江苏惠斯通电机有限 ...· 9 月前 · |
|
|
奔跑的手链 · 《雪中悍刀行》打戏被嘲，慢动作当道，“武打” ...· 1 年前 · |
Mail Data Loss in macOS 10.15
I’m working on more posts about the Catalina release, but I wanted to start with a short warning. I’ve heard a bunch of reports of data loss in Apple Mail. Thankfully, none seem to be caused by my apps . (Ironically, one of the bugs I’ve encountered is the inability to delete messages via AppleScript.) And, in fact, most of the damage has occurred without my Mail plug-in even being installed. Nevertheless, people contact me because it’s not unreasonable to wonder if third-party software is to blame, and I also hear from people who want a second opinion because what Apple support told them didn’t make sense.
What I’m hearing:
Updating Mail’s data store from Mojave to Catalina sometimes says that it succeeded, but large numbers of messages turn out to be missing or incomplete.
Moving messages between mailboxes, both via drag-and-drop and AppleScript, can result in a blank message (only headers) on the Mac. If the message was moved to a server mailbox, other devices see the message as deleted. And eventually this syncs back to the first Mac, where the message disappears as well.
I don’t know whether these are due to Mail bugs or to other factors such as problems on the Mac or with the mail server. But my advice is to hold off on updating to Catalina for now. These sorts of issues are pernicious because:
You may not notice that anything is wrong unless you are looking at the particular mailbox or messages that are affected.
Because the data is synced to the server, problems can propagate to other Macs and iOS devices.
Making a backup is difficult because, even if you set the preference, Mail no longer automatically fully downloads all messages. So the backup of the local data will necessarily be incomplete. (See EagleFiler’s Importing Attachments instructions for more about this. I’m happy to have most of my mail archived outside of Mail.)
Of course, it’s good to make backups anyway.
Apple advisors are apparently telling customers that if Mail data gets lost on Catalina, it can’t be recovered from a Time Machine backup that was made using Mojave. This didn’t make sense to me, and I’ve verified that it’s not the case. You can use Time Machine to get at previous versions of the folders in Mail’s data store, and then use the File ‣ Import Mailboxes… command to selectively import them into Catalina Mail. Since they import as new, local mailboxes, this shouldn’t affect messages that are on the server.
I also think that the advice to restore the whole Mac to Mojave makes no sense because as soon as you launch Mail it’s going to delete all the messages that were deleted on the server. In order to actually restore them, you have to make copies of the messages that might have already been deleted. That’s what Import Mailboxes does.
Update (2019-10-11): See also: Howard Oakley , TidBITS , Hacker News .
Update (2019-10-12): See also: MacRumors , iMore .
Update (2019-10-13): See also: AppleInsider .
Update (2019-10-14): Commenter Shiner says that the lost messages from “updating Mail’s data store” happens for local mailboxes every time Mail’s database is rebuilt, not just when migrating from Mojave, and reported this to Apple in August. It’s “trivial to reproduce” and lost 160,000 out of 200,000 messages in a test.
There’s some evidence that the moving messages issue is related to Microsoft Exchange accounts.
A doctor friend of mine lost about 20.000 emails when he upgraded to iOS 13. It was a POP3 account.
See also: TechRadar .
Update (2019-10-15): I’ve received four reports that messages moved to an iCloud mailbox (that were not originally in that iCloud account) simply disappear instead of showing up there. At first, we thought this was a Catalina issue, but it also seems to be happening with High Sierra now. My guess is that it’s an issue with certain of the iCloud servers.
Update (2020-01-29): macOS 10.15.3 fixes one of the bugs but not the others.
Update (2020-05-27): macOS 10.15.5 has the same version of Mail as macOS 10.15.4 , which did not include any further fixes, as far as I know.
Update (2020-07-16): macOS 10.15.6 has a new version of Mail. I’ve received several reports that moving iCloud messages no longer makes them disappear, but there is some reason to believe that this may be due to a fix on the iCloud server rather than in Mail itself, meaning that the problem may still occur with other mail providers. If you were seeing messages disappear on previous versions of macOS 10.15 and have updated to 10.15.6, please post in the comments about which type of server you have and whether you’re still seeing the bug.
Update (2020-08-03): One commenter reports that the message moving bug is not fixed in macOS 10.15.6.
Update (2020-08-09): I received a report of the message moving bug on the Big Sur beta.
Update (2020-08-18): I received a report that Mail in macOS 10.15.6 is deleting messages from multiple servers without downloading them.
Update (2020-09-08): I received two more reports that moving IMAP messages in macOS 10.15.6 instead deletes them. Several other users are reporting that dragged messages don’t get deleted but don’t move, either.
Update (2020-09-11): One user reported losing the entire contents of two IMAP mailboxes when updating from macOS 10.14 to macOS 10.15.6. I also received a report from a user who is losing messages when dragging and dropping them on macOS 10.15.6.
Update (2020-09-28): One reader reports that the IMAP moving bug is not fixed in macOS 10.15.7 .
Update (2020-10-07): I have received additional reports that the IMAP moving bug is not fixed in macOS 10.15.7 or in the current macOS 11 beta.
Update (2020-12-02): I have received additional reports that IMAP messages disappear when dragged and dropped between mailboxes (particularly when stored in different locations) on macOS 11.0.1.
Update (2021-05-25): I’ve received multiple reports that this still happens on macOS 11.3.1.
Update (2021-05-28): I’ve received a report that this still happens on macOS 11.4.
Update (2022-04-12): I’ve received a report that this still happens on macOS 12.3.1.
Update (2022-04-22): A reader reports losing messages moved server-to-local on macOS 12.3.1.
Update (2022-05-19): I’ve received another report of losing Exchange messages when moving them on macOS 12.3.1, and I saw lots of messages disappear when moving them from local to IMAP on macOS 12.4.
Update (2022-08-11): I’m still receiving reports of messages disappearing when dragged and dropped on macOS 12.4.
Update (2023-01-02): One commenter reports that moving messages from IMAP to local turns them blank on macOS 13.1.
Update (2023-01-18): A reader reports thousands of messages lost from On My Mac mailboxes with macOS 12.6.2.
Update (2023-01-31): Commenter Andy Scott is seeing data loss when moving messages between Exchange and local mailboxes on macOS 13.2.
Update (2023-06-19): Scott Palmer is seeing messages lost when moved out of an IMAP account on macOS 13.4.
Update (2023-06-28): I just encountered the bug again myself with macOS 13.4.1. When re-applying a rule that would move messages to an IMAP mailbox that they were already in, it just deleted them.
Update (2023-11-15): Some readers and customers have reported still seeing the data loss bug when moving messages on macOS 14.1.1.
Apple Mail Apple Software Quality AppleCare AppleScript Backup Bug Datacide E-mail Client EagleFiler Gmail IMAP iOS iOS 13 Mac macOS 10.15 Catalina Microsoft Exchange MobileMail SpamSieve Time Machine Top Posts
October 11, 2019 9:57 PM
@Morgan (1) I regularly archive messages to EagleFiler , which I wrote partially for this purpose. It converts Mail’s data store to standard mbox files and checksums them so their integrity can be verified later. (2) For messages that I don’t actually need to access across multiple devices, I have Mail rules that move them to local mailboxes (“On My Mac”). This ensures that there’s a complete local copy (with all the attachments) that will make it into the regular backups of my Mac. (This only works when the messages are processed by rules as they arrive. If you manually move messages to local mailboxes later, even if via rules, that does not force Mail to fully download them; it can actually lead to data loss as Mail saves the incomplete message locally and deletes the complete copy that’s on the server.)
The main issue I and others have noticed is that Mail.app shows blank Sent folder/mailbox from GMail/GSuite accounts. Here is a thread on the issue from other users:
https://support.google.com/mail/thread/16312735?hl=en
There's also Mail Archiver X, which can archive emails either directly from IMAP or from the local Mail.app's database.
I lost a lot of email messages when upgrading to Catalina. I would not have noticed except that some of my email accounts - which, apart from my Apple ID accounts - are all POP accounts so are not on any remote servers, seemed to have a surprisingly small number of messages after upgrading. I have a total of more than 100,000 messages going back about 15 years. On checking I found that some of my accounts were missing thousands of messages. Which was obviously very worrying.
Luckily I had backed up before upgrading and I painstakingly imported the missing emails from my backup. It took many hours and wasn't altogether successful because some sent emails are now logged not by recipient but by sender (me) which is not very useful. But I think/ hope I managed to find almost all the missing emails.
I am not a Mac expert but I noticed that after upgrading Catalina had created a new folder: user/library/mail/V7 which was not in Mojave, where my messages are now stored.
Perhaps this has something to do with the problem?
When the users move from v6 to v7 would the size of the folder be very different than before?
Import Mailboxes isn't a good idea because you would end up with even more duplicate emails than you have anyways in Mail.
The problems with deleting emails with AppleScript isn't really new. You can't even check if the deletion went okay or not.
[…] via mjtsai […]
There is a bug in Mail under iPadOS 13.1.2. If you send an email to yourself (say to read on a different device), it will appear in your Inbox and in your Sent box, as expected. If you move the copy in your Sent box to the Trash and then Delete it, the copy in your Inbox is deleted, too (and from your other device). That didn’t happen under iOS 12.4.
@Sam Each major new version of Mail migrates to a new folder, i.e. V6 on Mojave and V7 on Catalina. So that’s normal, and it’s good because it keeps the different versions of the data store separate.
SuperDuper is great, but I think it doesn’t fully support Catalina yet.
@Beatrix I think the file format hasn’t changed much this year (whereas a few years back the way attachments were stored changed). So in theory the sizes should be similar. However, I have also seen (going back a long time) that sometimes when the data store is rebuilt it throws out all the local copies of messages that are on the server. So in that case the V7 would temporarily be much smaller.
Yes, duplicates are a problem (#4), but that can be fixed. It’s much better to have duplicates than to lose your mail.
Deleting e-mails via AppleScript has always had issues, but it’s gone from working something like 99% of the time to failing 99% of the time.
I routinely go through the pain of manually exporting all of my e-mail off Mail.app onto .mbox to prevent this kind of thing, since ever since they started messing up .emlx and storing attachments separately that the SQLite metadata stores have had a number of issues.
I suspect that in many cases mail messages are indexed by something else than the Message-ID header (which is in itself prone to duplicates) and that the indexes get confused, but overall (and having used Mail.app since the NeXT days), whatever has happened this time around is an(other) egregious failure of QA at Apple. There simply is no excuse for the amount of bugs that have shipped in macOS (and iOS), and for someone like me (with an e-mail folder tree that goes back to 1998) this means I won't be upgrading to Catalina until they sort it out.
(Ironically, I am more likely to move to an iPad Pro full-time for my personal stuff and set up a Windows or Linux box to work - I have a lot more trust on either of them to deliver stable releases right now...)
@Rui Unless it’s been fixed in 10.15, the built-in export from Mail to mbox is problematic because it doesn’t properly escape the “From ” lines (so message boundaries will get messed up if you import into anything other than Mail, which reads the boundaries from sidecar metadata) and it doesn’t download attachments that are not already cached locally. EagleFiler avoids this by generating the mbox itself and checking that all the attachments are there, and it also saves the metadata that mbox doesn’t support.
[…] dans cette même application. Il a expliqué la situation détaillée dans un article posté sur son blog : apparemment, un nombre non négligeable d’utilisateurs a rapporté qu’une mise à jour des […]
Have you waited for one or two days after the upgrade? Have you tried re-indexing with Spotlight?
I had problems with my 60.000 items mailbox almost during every MacOS upgrade before; and the symptom was missing emails. In the end, I suspected it has to do with Spotlight: The list of mails in mail is created using a Spotlight query, and upgrading mail.app leads to re-indexing all mails in Spotlight - which takes time (and there is *no kind if indication from MacOS which is really a pain*). During that indexing, a lot of mails will simply be missing. In particular tough cases I launched re-indexing with Spotlight (forgot how to do this, but you‘ll find it easily) and that helped (and you can see whether it is finished by looking into Activity Monitor).
Can‘t say yet anything about Catalina, but it would be interesting to know if anyone who already dared to do this can solve it in this way?
@Michael yeah, I know about the "From" thing (I maintain https://github.com/rcarmo/imapbackup ), but I was under the impression mbox export would download attachments (since I have that option set to "always") even if they weren't extracted from the MIME info (or even downloaded already).
Can I take an EagleFiler archive and reconstitute a full IMAP tree from it?
I upgraded and then saw this thread! I have not opened the Mail program yet, and was just about to re-install Mojave. But before I do, I wonder if I use POP accounts (still) on my own company‘s server, whether I’m still at risk of losing emails.
Any clues?
@Sarah
I have not heard of any problems with POP accounts, although that could simply be because most people don’t use POP these days. At least with POP you can make complete backups and don’t have to worry about corruption getting propagated to other devices.
Commenter Sam
has lost messages in POP accounts during migration. I’ve not heard of any POP messages lost moving them between mailboxes.
@MichealTsai @PeterResele From my experience with the innards of Mail (and I have a fair bit after working on plugins for 15 years), both regular mailboxes and spotlight mailboxes are populated by performing a spotlight search to find library ids (ie, the integer part of the .emlx filename) and then looking up those messages in internal sqlite database. Mail also uses spotlight for doing the message counts for mailboxes.
Given this, rebuilding spotlight indexes may resolve the missing messages issues for some but mileage may vary.
As an aside while I personally think this is inane because Mail has a highly reliable, performant sqlite database, I think Mail uses spotlight to prevent a discrepancy in which a person will do a system wide spotlight search for a message will turn up the same result as a search inside Mail, and a few other reasons.
@Scott I think we’ve both been writing plug-ins for Mail since before it was using SQLite and before Spotlight existed. :-) Yours certainly hook into it more deeply, though. It’s an intriguing theory. I don’t know what to say, because it doesn’t square with my experience successfully using Mail with Spotlight completely turned off, the fact that Mail’s SQLite database has fields and triggers to maintain message counts, or the fact that I’ve fixed messed up counts by editing the SQLite database. I’ve also wondered why they don’t they don’t use Spotlight only for the full content searches, given the performance of SQLite.
I'm wondering that all the smart people here dont know how to use Time Machine properly. You have to be in Mail and start TimeMachine from there. Now you can restore single Mails or folders directly. No importing Mailboxes from the Library.
Please correct your post.
[…] El más reciente, reportado por Michael Tsai, desarrollador de EagleFiler y SpamSieve, un complemento para la app Mail de macOS, asegura que el gestor de correo electrónico por defecto de Apple puede provocar pérdida de información si se actualiza a macOS Catalina, según ha comunicado a través de su blog personal. […]
Just a report here for other users:
(1) I have not noted any loss of email, but I noticed today that one message did not have its attachment downloaded.
(2) Search stopped working -- it's really annoying. I have had to go to the office.com outlook app to search for
emails, nothing is found on mac mail (it just shows that it is "searching"). I am currently rebuilding one of my mailboxes
to see if it gets fixed.
[…] وحدد مايكل تساي Michael Tsai، الذي يطور إضافات EagleFiler و SpamSieve لتطبيق آبل للبريد الإلكتروني Apple Mail على نظام التشغيل MacOS، بعض المشكلات التي ظهرت في تدوينة. […]
[…] upgrading to macOS Catalina. If you depend on Mail for your work or in your personal life, please, read this before upgrading. […]
I have an open ticket on this going back to August (and was probably the first to raise the alarm on this one.)
Where you will see this issue is with "On My Mac" mail. If the Mail application launches with the "Mail Message Import" box (that you can't ignore), when Mail does it's "reimport" of all your mail, you will most likely have a number of "On My Mac" messages just flat out deleted. This is new behavior in 10.15 and wasn't taken seriously enough (IMO) when I reported the problem.
It's trivial to reproduce (if you have backups!) by deleting the "Envelope Index" files (Google around for how to do this -- it's been a long-standing recommended way to fix "search" issues *within* Mail), . It's not related to moving the Mail store from "V6" to "V7", and shouldn't (at least in any of my testing) affect IMAP mailboxes (as those just refresh by pulling mail back down from the server.)
I have test mailboxes with 200,000 .emlx files that end up with only 40,000 .emlx files in them after Mail does it's reimport. If you have "On My Mac" mail, you should test this yourself and immediately file feedback on it!
@Peter It does seem like Outlook/Exchange is overrepresented among the people with the moving messages problem. But there have also been multiple reports (for a different blank messages issue) with Gmail. And the importing issue seems to affect IMAP accounts and even local mailboxes.
@Jörg Have you tried using Time Machine from within Mail on macOS 10.15? Every user I’ve talked to said that this caused Mail to crash, as it did for me. It’s also not clear to me that Mail Time Machine can restore mailboxes that were backed up from previous versions of Mail.
@Shriner That’s really bad news if it’s related to rebuilding the Envelope Index database rather than just migration. There are so many problems that can only be fixed by rebuilding, which has historically been very safe. Even if IMAP messages can be pulled down from the server again, losing the local files in a rebuild would discard information that doesn’t sync to the IMAP server, such as colors.
[…] Developer Michael Tsai shares reports on users with Mail problems in macOS Catalina specifically missing messages or incomplete messages. In some cases you will see headers only and a blank message. Tsai recommends not updating to macOS Catalina yet, but if you already did he shares one way to fix this issue. […]
Yeah, I've never seen this in all my testing that it's related to migration *unless* the Migration forces the reimport (which, I believe, shouldn't happen intentionally.). I only noticed this because one migration I did forced this reimport and I didn't realize I had lost mail for a week or so after that -- the mailboxes (as seen in Mail) will still show the same number of messages -- it's only when you go to open them that you find they open blank (and then you can see the .emlx files are gone...)
My gut feeling is that people who have upgraded to 10.15 have already lost a lot of e-mail, but don't realize this yet.
[…] y SpamSieve, complementos para la ‘app’ Mail de macOS, ha difundido a través de su blog que ha recibido varios informes de usuarios de macOS Catalina que afirman haber sufrido una […]
[…] part, elle contient quelques bugs très gênants, dont l’un d’eux rapportés par le blogger Michael Tsai, concernant les e-mails et plus spécifiquement, l’app […]
Michael, thanks for this, very helpful.
I'm not sure if you are aware of another even more serious mail issue that a number of us are hitting, which causes Mail to be completely unusable.
The error message you get is: "Mail cannot save information about your mailboxes because there isn't enough space in your home folder" ... the popup with this message has only one option, Quit. The popup happens some period of time after mail loads, sometimes almost immediately, sometimes after 1-2 hours.
This is an error that has appeared in the past with Mail but the fixes that worked in the past (removing index envelope data and the like) do not fix the issue this time, nor does reinstalling Catalina.
Here's the issue on Apple discussion forum: https://discussions.apple.com/thread/250717871
[…] Learn Extra […]
Same issue here. Just lost a couple hundred thousand mails, randomly, all over the place, from the last 20 years.
Am restoring, thanks to Mike Bombich, using Carbon Copy Cloner, from nightly backup, which I had first mirrored to yet another disk.
It's running since a night or so, and the mails are coming back, with their folder structure. Just take one by one the content of each subfolder in ~/Library/Mail, and import it using Mail's import function.
This problem went unnoticed for a day over here, since the mails appear to be there. It is not dependent on what type of mail server you had been using - I had been using all POP3, IMAP, Exchange over the years. The mails that were lost were all local to the machine, i.e., long time deleted from the (in many cases long time not longer existing) mail server.
So it is not just Online folders, it is everything, everywhere, with no apparent pattern.
Beware. And Apple - WTF?
[…] plug-in developer Michael Tsai has posted a warning on his blog to reveal that after installing macOS Catalina, some users could end up with missing or incomplete […]
[…] warning arrives by way of SpamSieve plug-in developer Michael Tsai. His post is based on feedback provided by customers, though the Mail problems aren’t related to […]
OK, here is an update on this topic:
I've first of all gone back to backup and restored all of them.
This morning, all ok.
This evening, WTF? Again skeleton mails from today.
I think I've found some (frightening) systematic:
Don't move a mail that has an attachment out of an online folder. It'll just kill the mail content. You can copy it into another online folder, even from like exchange to imap, but if you move it out, it seems to become a skeleton.
There's more inconsistency in that; sometimes it doesn't even perform the copy / move operation.
So, summary: DO NOT USE APPLE MAIL for the moment.
I have a 2018 Macbook Pro and I'm having search issues with the mail app: the query does not find all of the relevant emails. I've tried rebuilding and reindexing spotlight with no success. The only thing that I found that worked was to manually look for the email and view it. Only then, it would show up in the search query.
When I upgraded to Catalina I experienced a new problem now of not being able to email back to myself, which I do with many emails. They all bounce as a Permanent Error. I have complained to Apple for years that Mail was behind the times and had lots of bugs. Now we get this new set of problems. Search is just not a valid structure, it is hit and miss. Signatures are so awkward to set up. So much is not clear from the help menu. Things just do not work they way they say. Now with Catalina I see sliding lines of text and mail folder that I have no idea what they are to do. They could have put some info help boxes into this it seems.
And the date/time of the incoming email cannot be placed on the left side like I had it. It is locked on the right column. Nothing in View will give this option. I need it on the left side.
There are no ways to color icons either that might help users.
I have not noticed any lost emails with Catalina (may not have discovered it yet). I had this problem long ago with Mojave back in the spring and tried to alert Apple. I lost 15 years worth of emails when I migrated to my new MacBook Pro in January. The MBP had a lot of problem root software they had to find and take out. In the process I lost the emails when he set up a test user account and I then went back to my main user account. The tech told me I had put them on my Comcast ISP in the Mail mail boxes and they deleted them but I know I had them On My Mac mailbox using IMAP. They would not restore from Time machine either, so they told me.
Another problem has came up with the migration to the MBP, either Mail or Comcast has been deleting all my emails over one month old. At first I thought this was a preferences setting on the MBP, which had been put in by the migration but it even exists on Comast ISP with the MBP settings ok now. I called Comast and complained and they said they pulled all my old emails from an archive file and put back in my ISP mailbox but they all show as coming from me with my name, so that is not much of a help.
Long ago I found a big problem trying to delete incorrect emails from Previous Recipients. They just will not go away and keep showing up in the TO field while not in the archive list. AND for a long time Mail has been deleting email addresses of people I may not have mailed in a year or so. They are just gone, removed, I did not remove them.
This is a sorry application for such a pretty good computer company, imo. Now I am new to these forums and don't know how to get replies or other info...but if someone wanted to help me send an email to me at citizen
@comcast.net. How does one email Michael Tsai directly?
Thanks,
The problem I have experienced since updating is that emails that I know have been sent to me, and I can verify were sent, are not appearing in my inbox. Yet, if I check with the sender and know the Subject, I can find them when I search Mail. Effectively, the incoming emails are - apparently randomly - invisible unless I know they have been sent and I know how to search for them. They are in the inbox, in the right place, and they are visible upon a search, but disappear again when the mailbox is revisited. This is a disaster, and I understand from colleagues they have experienced similar issues. Furthermore, the whole Mail system is unaccountably incredibly slow now, every time I log into it.
My problem with the catalina upgrade on my 2018 macbook air is with the undo delete message option. It no longer works, and i actually use it a lot. Also, with some messages, but not all, the scroll down option does not work. Little annoying glitches! and I haven't seen the option of 10.15.1 yet. Also apple has not responded to my questions. I guess they have bigger problems than mine.
Yes, I too have experienced similar frustrating issues with Mail in Catalina. One additional problem, I typically move old mail from my sent folder to an archive of mail in organized folders on icloud. So, as others have said copying mail to icloud is just failing like crazy usually (for me) with a timeout error. The mail that failed to copy to the icloud folder has gone from its original folder and now shows up in a new "On My Mac" recovered mail folder. But the date for all of the mail items in that recovered mail folder in column view is now today's day regardless of when it was sent!!! The date in the mailheader is the original correct date. So if I select date with the new sorting dropdown menu the newly moved items are sorted with todays sent mail.
I guess that I just cannot move any mail until Apple decides to fix this mess.
Hi all,
My upgrade to Catalina on my late 2015 iMac went reasonably smoothly, until I logged into the 2nd User Profile on the machine, and saw an error message about having to import email mailboxes. Didn't get to note down the message, however, a brief synopsis of the issue & resolution is:
Ran upgrade to Catalina on Sat Oct 19.
Emails from a POP3 mailbox were randomly missing content in the Mac Mail client, from a time period of Oct 18 back to April 2018.
By necessity, the emails had been deleted from the mail server - lack of provisioned space.
The mailbox had been in existence for about 12 months, but the last 7 months or so of email had only headers, no content.
The emails from the last couple of days were valid with headers & content.
Upon further investigation, and rebuilding the mailbox a couple of times, the emails without content disappeared entirely. I could verify this, as MS Outlook had been setup and had downloaded a copy of each mail item into its own mail store. However it did *not* contain all the sent emails, and was not gong to be solution.
Found the local mail store on the filesystem, went into a Time Machine backup point just prior to the upgrade being run. Restored the entire /Users//Library/Mail/V6 (Mojave) to a temporary directory in Catalina.
Imported *all* the mail into a new folder in Mail application. Was given the option to select a number of different folders, but chose bring all mail items in.
Found all the missing content, dragged & dropped into correct folders, including Sent emails for missing time period within the Mail app.
Seemed to have fixed the problem, however will require further checking over the next day or so.
I've been seeing "strange" Mail issues (okay, mostly using GMail backed mail accounts) even before I considered Mojave... that was on !0.12 & 10.13... where my laptop was still on 10.12 and my desktop on 10.13, I did on several occasions had to "rebuild" my mail database as I experienced (especially older) mails without their attachments or just empty.
Lately, after I've "upgraded" the desktop to new hardware + Mojave, I've been encountering (on both the laptop & the desktop) emails that are wrongly put in the same mail thread ;( some old non-related included in the same one ;(
Also discovered that the majority of my local mailboxes (hundreds of hierarchical mailboxes, >500,000 messages) had only partially made the conversion to the V7 mail store. There were missing messages and no body or attachments for most of the messages that were there. I had no problems with the four Gmail accounts syncing. I restored the V6 folder from right before the upgrade to 10.15 using Time Machine in the Finder, then imported the local mailbox folders into Mail piecemeal starting with the top level hierarchical folders. If you select a mailbox folder for import directly, you will also get the UUID folder names it contains, so put the mailbox in a folder and select the containing folder if you want to import individual mailboxes. Consolidation with old folders is going to be fun.
Some comments relative to the main post - no Exchange accounts were involved. There were none defined in Internet Accounts at the time of the upgrade.
Search wasn't working at all, but rebuilding the spotlight index didn't help in this case (there were missing message files - i.e. no underlying emlx files and attachments), and rebuilding the Envelope Index did make things worse (fewer messages listed and body no longer showing up for most messages that it had been visible in list preview before rebuild).
I had issues when upgrading to High Sierra with losing some old archived email, but the Catalina update really destroyed my email. 20 years of emails, archived in folders by year, and most of them are blank. Apple was unable to help. While I have backups, they are imperfect since the issues started with the last OS. I am ready to throw in the towel and setup all new email addresses.
These are the test I've done with results;
Like most users have reported, I too lost a huge amount of emails by updating from Mojave to Catalina. I had a solid TimeMachine backup so I simply formatted my drive then installed back Mojave and restore from TM. That worked perfectly and all my mail was back the way it should be. I have an extra MacBook Pro 13" so I decided to do some more testing. I first created a new folder on my Desktop and exported all my mailbox via the export function. Then I did a clean install of Catalina on my 13", open the mail app and imported all my mailbox via the import function. This allowed me to re-create a perfect copy of mac mail Mojave to mac mail Catalina. Then I made a brand new TimeMachine backup of my 13" Catalina install. Now I'm thinking I have a TimeMachine Catalina backup with all my mail so that should be perfect! Well no it was not. I erase my 13" and proceed to install a clean copy of Catalina, then use my perfect TimeMachine backup to restore thinking that my mail should be back the way i left it. Believe it or not, upon restart and opening the mail app, most of my mail only show headers and nothing in the message. This is a terrible bug and looking at the release notes of up-coming updates, it looks like Apple is not even addressing this issue! This is terrible really! Bad Apple :(
Hello,
I have lost all the content of an outbox mail of one of my gmail account when upgrading to Catalina.
I was in contact with Apple international support via telephone and they have confirmed me that this issue is known to them as a bug in Catalina, which will be fixed in the future.
I suppose that more and more applying to them would accelerate probably their speed.
@Michael Tsai Of course I tried "apple mail" option too but lots of errors in console with newer ending story of import. What about "Mail Library cannot be read because its format requires a newer version of macOS." is there solution to convince Mojave version to read Mail folder which was already in use by Catalina?
I am trying to solve 120GB emails which are present in Mail folder but not shown in Catalina, so I moved to Mojave and trying to get those emails in app...
@Miloš I had thought Mojave Mail would be able to import from Catalina Mail directly. If not, you could use my EagleFiler app to convert the files. Make a new EagleFiler library and then drag and drop all the Catalina account (UUID) folders inside the V7 folder into the EagleFiler window to import the whole mailbox hierarchy. This will convert all the mailboxes to mbox format. Then you can tell Mojave Mail to import the mbox files stored in the EagleFiler library.
I did a clean install of Catalina 10.15.1 today (Nov 1) and the bug is still not fixed.
All of my inbox messages (POP mail) from the past two years vanished sometime after I did the install. And for others, message headers are showing "no recipient" "no subject" and content is blank.
The problem persists in 10.15.1. Almost 50% of my old emails have now disappeared (40Gb of info). Most of the emails only show headers and nothing in the message. When rebuilding the mailboxes, the emails just evaporate. I've looked into the finder structure and the emails are completely gone. I can only recover from backups, but many emails will be gone forever. Bad.
How can Apple release this with such a bug? It means you're taking a risk updating to Catalina, and one more serious than usual, because the data loss can propagate to your other devices.
It all seems odd - all that marketing for the new and great macOS, yet Apple must surely know that 'upgraders' are risking losing data! Unbelievable!
Might this blog thread make a note if and when the problem is resolved?
It is ridiculous that Apple would allow this to go ob for nearly a month. We are losing all corporate emails that have attachments (85%) as the email are blank once pulled onto our servers and personal computers. We do not store email on our server, they store locally and this a significant issue for our business.
We have worked on every possible fix and there is NO WORK AROUND to recover blank emails nor to stop the emails from converting to blank documents.
downloaded Catalina two days ago, went to check email and a blank email page opens without any listing of any emails. Mailboxes shows the various mailboxes but there is no information or display when you try them. Not sure where to turn for any fixes. My email is cloud based. Luckily email still shows on iPad mini and iPhone X S.
I think you guys have just helped me solve the big issue I have been trying to fix for the last three days. I upgraded to Catalina last week and have since lost 3 very important drafts in my Apple mail...they show up as drafts (which they are) and I can see the the preview with the first three lines of text in the drafts folder on my iphone Apple Mail App but when I open them, they are blank. None of these drafts show up in Apple Mail on my Mac, nor in Gmail. They simply never synced it appears. Sadly, one of the drafts was SUPER important to me, and now it appears corrupted and unsalveagable. I spent an hour on the phone with both Apple and Google support but now I think I understand the problem. I will hold on to the draft in in the draft box and hope that once there is a bug fix it comes back to life, but I am losing hope. I really need that draft. Any advice?
I experience the same problem. When I move or copy emails received from my outlook exchange account in another folder in Apple Mail, they appear to be totally empty, despite the fact that the sender, subject and date/hour appear correctly. You have to open the mails to realize the problem. The problem has appeared since the upgrade to Catalina. Other accounts (Gmail) are not affected by this problem. Only mails with attachments are affected. Other mails can be moved or copied without problem. Another problem is the non-synchronisation of the outlook exchange account. Mails read or deleted in Apple Mail do not change status on the Exchange server. As a consequence they appear to be unread or undeleted on my i-phone. This second problem seem to affect all emails in the outlook exchange account, with or without attachement. My solution (well not a real one, more a safety trick) as of today is to keep all mails from my outlook account in the inbox and postpone the cleaning until the bug fix is released.
This also happens with POP accounts in Apple Mail in High Sierra (10.13)(I don't use iCloud, so it's not just an iCloud problem with High Sierra). I first noticed it this summer: I couldn't find a message that I was fairly sure I had saved (all my saved messages are in a folder named Archive). A few weeks later, I looked for a different message that I KNEW I saved and that wasn't there either. When I looked at my Archive folder, it only had about 600 messages; there should be 2,000 messages. Restoring from Time Machine backups didn't work (every time I imported the TM mailboxes into Mail, the number of messages didn't change). The messages started disappearing around June 2019 (this is when the message counts in the TM mailboxes started going down).
What I finally did was use EagleFiler and imported every mailbox from every TM backup I had (yes, it took a lot of time, but I was able to restore every message, since I had TM backups going back to April). Now, I export my "Saved Messages" folder from Apple Mail to a .mbox file every few weeks and add it to my EagleFiler library, so I don't have to worry about losing messages.
THANK YOU, Michael, for creating EagleFiler, you saved my bacon.
If you really need to move a mail from the Inbox (from an Excchange account) to another Apple mail folder, you can copy the message on the Desktop, then open the message from the Desktop and copy it to the target folder. Doing it, the message is copied correctly. Do not forget to delete the message from the Inbox if you wanted to move it and not just to copy it. Obviously, such a procedure is far from being user friendly. At the very least, it shows to the Apple developers a step-by-step procedure that does not transform Exchange messages into blank messages. I leave it to them to make it automatic;-)
Question: when does the data store upgrade happen - on installation of Catalina or on initial launch of Mail app? I installed Catalina a few weeks ago but happened to read this thread before opening Mail - wondering if the damage is already done, or if I can wait to open Mail once Apple issues a fix. (reading mail via iOS, fyi) Thanks!
[…] Mail trên macOS, đã phác thảo một số vấn đề đã được khắc phục trong một bài viết trên blog – ông khuyên nên giữ bản nâng cấp Catalina, nếu bạn dựa vào Apple Mail, cho […]
I made a clean install of Catalina (booting from usb stick containing 10.15.1) deleting Macintosh HD with disk-utility, then installing the operating system) and recovered my profiles via Time Machine. Locally stored emails and emails in pop accounts appeared blank except for the email header. I tried this several times. Then I made a clean install of Mojave and recovered my profile from the same time machine back up, and everything is fine. In other words, the problem is not a potentially corrupted back-up, but OS X 10.15.1
Each process about six hours, so I’m leaving it until fall 2020. CONGRATS APPLE!
Hi All,
I have been spending about 9 hours with an "APPLE expert" by the customer care service in Italy to try diagnose and fix the problems induced by Catalina on the mail client. No way, after every iteration it got worse. By now I feel like saying it looks very much related to the interaction with Exchange servers. Something got really screwed up. Back to the old story between Microsoft and Apple. I reported to the "expert" this is really something that should not happen; it makes no sense to release a new operating system without these very basic tests. I'm really disappointed.
Michael, if I upgrade to Catalina and I don't have any interaction with Exchange Servers but only Gmail and Fastweb (an Italian server) accounts, do you think I could have problems?
Another question: if I no more use Mail but another mal client (i.e. Spark), could I avoid mail problems upgrading to Catalina?
Thank you.
Just to keep everyone informed: macOS 10.15.2 Beta 2 comes with a slightly fixed version of Mail.
10.15.1: Mail 13.0 (3601.0.10)
10.15.2 Beta 1: Mail 13.0 (3608.2)
10.15.2 Beta 2: Mail 13.0 (3608.40.2.2.1)
I don't know if anything has been fixed since I don't have any Mails in my Beta-VM but I wanted you to know.
I can neither confirm nor deny that the "envelope index rebuild mail deletion bug" is *NOT FIXED* in any of the 10.15.2 betas yet.
So I can not confirm or deny that you *shouldn't expect it to be fixed in 10.15.2 at all* at this point because Apple said that a "lack of duplicate reports" is not moving this higher up the regression fix priority list.
I can neither confirm nor deny that we are not pleased about this by any means.
I *can* say -- that if you have tested this and you have reproduced the problem and you have not filed an official bug report about your results -- this is why it's not fixed yet. :-(
Im wondering if anyone can help me ...since upgrading to catalina , any time i begin to compose a message using mac mail but using my out bound webmail address , i receive this message below, it freezes my mail and I have to force quit.... the actual email server works when I log in through the website and sends the email zero
problem
The message “Re: ” could not be moved to the mailbox “WEBSITE Email” ...
Mail was unable to open this mailbox on the server “mail.lngphoto.com”.
The server returned the error: Mailbox does not exist, or must be subscribed to.
@Lisa Greenfield
At this point I would not trust anything Mac mail using Catalina. If you're able to revert to Mojave without any lost of data, that be the road I would take. It's shameful what Apple is doing here. Tempering with precious emails in that way is clearly unacceptable. It's almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!
I wonder just how widespread these issues might be given Apple's response to Shriner. I migrated two 2016 15" MacBook Pros to two new 2019 16"MacBook Pros that had Catalina 15.1 preinstalled last weekend without any apparent Apple Mail related issues. It's been 3-4 days since I did the migration to the new Macs and have done routine spot checks of my mail & attachments without finding any issues. I DID import all current mail (~26,000 emails on one machine) into EagleFiler, after confirming download of all attachments, as a precaution.
Note that neither new Mac did a mail "reimport" to my knowledge and I have not attempted to rebuild envelope indexes. I have three imap accounts (Gmail, iCloud, and one associated with a personal website) although the vast majority of mail is in the iCloud account. I also have an extensive "on my mac" folder hierarchy. I do not have an exchange account or utilize Outlook. So far, so good but I'm still spot checking and hanging on to the EagleFiler databases!
I'm fed up with this. I'm importing all my mails in EagleFiler. Hey, how can I change the font size in EagleFiler's Mails and Lists?
The way I'm importing is to work through a local dovecot that I've dockerized.
It’s almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!For whatever it’s worth, I’m not running into the issue.
My guess is this particular issue was non-trivial for Apple to reproduce at the time, and didn’t see quite enough users affected to make that high-priority enough to delay the entire release. (Which, it seems, wouldn’t have helped anyway, as they still haven’t been able to fix it by 10.15.2.)
The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing. :-(
That can cause serious problems when the email is needed later for proof.
Frank
Hello,
Disabling Mail Preferences --> Rules appears to solve the issue of receiving seemingly random empty messages into my Exchange accounts (IMAP), which I had migrated from an earlier version of OS X.
Hope this is useful.
I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again.
I've contacted AppleCare, showed them the problem, they collected info from my machine, and now I'm awaiting a reply.
I also encountered problems with email after upgrading to Catalina. On the phone with Apple for hours, the guy telling me they'd had a lot of problems with this update, and they hadn't really tested it properly. I rolled back to Mojave. Now wondering whether they are ever going to address the issue. Any recommendations for an email client I can use instead of Mac Mail, where I can import all my emails and settings, anyone?
Yes, I can confirm that after weeks, I still experience *serious* Mail issues (made the mistake of installing Catalina from Day One and then hoped for updates, all installed). It's now "better" than at the beginning, but still totally bad.
In addition to the points mentioned previously (pasted below), I also have to manually delete everything in Mail.app (iPhone), so any spam I delete in Mail (MacBook), I must delete twice there plus once in the app…
from Frank:
"The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing."
from Robert H:
"I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again."
Is there any indication that MacOS Catalina 10.15.2 Beta 3 will address the critical problem of lost emails bug(s)?
Like others I will finally migrate my 'archive' of emails to FileEagle librarys once I see the path to reconstructing what Catalina has destroyed.
@ John Hart: From Catalina to Catalina. But the from-Mail was completely screwed (see my first comment above). I went from my old MacBook Pro which was already on its last legs, had Catalina, and experienced all system updates for the past 6 years to the Library folder (Finder: press Option > Go > Library) and simply copied the Mail folder on an external drive. I then went to my brand-new MacBook Pro which I set up from scratch (NO migration), opened Mail.app there, and imported the Mail folder from the drive (File > Import Mailboxes: Apple Mail). Hey presto, took a while (220K emails) but it looks nice now. :-)
Kakis so there is no need for exporting Mailboxes in .mbox format? Will this technique work when going from Mojave to Catalina? I used to only copy Mail Library folder on older systems, but it stopped working few OS systems back. Are you saying I can copy my Library folder to external HD and import this very folder in Catalina and all my mailboxes, rules , etc will be copied?
Just got bit. After upgrading, all was fine. Then I tried fixing an issue with a smart mailbox displaying that there were messages in the mailbox when it was empty. Boom, hosed. My accounts are all POP (except iCloud and Gmail, which I don't really use but we're all forced to have). As others have said, only the large boxes are affected. I can't say how large they need to be, but my boxes with Amazon receipts, my wife's emails (going back to when we first met in 2002), and base inboxes full of unfiled messages are toast. If I go into Time Machine, they show up, but I'm not able to restore -- even from a TM disk I just started a few days ago. Diving into both the current V7 folder and ones from TM backups only nets me partial success. So far, the most recent message I can find in there are from three weeks ago. Both in TM and on Crashplan. Where is Mail storing more recent messages in the meantime? They're there somewhere. Also, new message since this happened are fine. So far.
@ Kakis: Did Mail rebuild the Envelope Index database? I can also copy my Library folder (V7) from my second, older Macbook and it looks and works fine. However, I'm not able to perform a single search. And as soon as I rebuild the database, all the data of folders containing +2k emails gets lost.
Can anyone else confirm a cutoff at around 2000 emails per folder?
I found a workaround for my system! After having bought a 2019 16" model last week, I upgraded my 2014 Macbook Pro from Mojave to Catalina and migrated all data to the 19" model. Mail worked fine for the 2014 model and was simultaneously heavily screwed up for the 2019 model as mentioned above - perhaps the one big difference being that after migration the latter entirely rebuilt its Mail database, whereas my old Macbook apparently didn't. I tested that a few times, deleting the current V7 folder (which would go from 6GB to "Import Mailboxes", and then selecting "Apple Mail" rather than only chosing particular mailboxes. After importing the intact V7 folder from my older Macbook this way, all messages showed up correctly and were searchable. The import does jumble up the order of folders and mailboxes, but that can easily be sorted out manually. I then imported signatures and rules and have a working Mail App without any data loss so far. Hope it stays that way and hope this is helpful for someone out there!
And btw, just to be safe I installed EagleFiler which does an excellent job in archiving all or any mailboxes and folders.
I don't know what to say other than what I wrote before (I'm not a computer person). As I said, I had thousands of emails, some folders also had more than 2k (to answer one query above). All I did was copy the Mail folder in my library -- but at that time, the old laptop was already in Catalina (so there weren't any V7 folders any longer). The main thing is that when I updated my old MacBook Pro to Catalina, all hell broke loose and Mail was rubbish with all attachments gone/not displaying -- and when I coped the Mail folder from that faulty laptop onto the new MacBook Pro (clean, no migration but with Catalina), everything was fine.
@Shriner, I am having the same issue with all of my POP and IMAP accounts except for my iCloud account and I have filed FB7490433 with Apple. I don't use gmail, just POP and IMAP mailboxes from my hosting provider.
I note that prior to my Catalina migration my Mail folder had 5,792,038,039 bytes (5.82 GB on disk) for 101,388 items and after running the Mail app after migration I had 2,871,906,047 bytes (2.8 GB on disk) for 37,633 items in that folder. All of the message envelope meta data appears to be there, but as I go back in time a few weeks on some folders, a few months on others, the message bodies are all blank.
I was able to import all of the mailboxes from the good backup into AirMail 3 (and into Mail Steward as backup) and all of the data was there.
I’m posting just to underscore a point that Shriner made above but that I don’t think has really been emphasized enough (both here and elsewhere with this issue): the issue ONLY affects local “on my Mac” mailboxes.
After careful preparation – 3 separate backups, and pruning and archiving (to mbox files) of my mail – I just upgraded to Catalina 10.15.2 yesterday. I was somewhat surprised when I crossed my fingers, held my breath, and launched Mail fo the first time....
It just launched. There was absolutely no updating or conversion of any of my mail. It was no different than launching Mail on Mojave.
The key which I see now, but wasn’t clear to me then: I only have IMAP (and Exchange) mail folders, with nothing stored locally on my Mac.
I realize for many people this potential bug is still a deal-breaker,but for anyone out there who I has been waiting to upgrade and either (A) like me, only has IMAP mail, or (B) can move things around so that they only have IMAP mail – then for those folks there is no reason not to upgrade now.
@Sam There are multiple issues. The one about upgrading and rebuilding mailboxes seems to only apply to On My Mac mailboxes. However, other users are having IMAP and Exchange messages disappear when moving them between mailboxes (even on 10.15.2). This doesn't seem to affect most users, but since we don’t know who will be affected I can’t recommend updating in general.
@Michael. Thanks. What you say makes sense.
It does make me wonder, though. It sounds like the "on my mac" problem has a clear source in Catalina: rebuilding the mail data store and/or the envelope index in Catalina Mail leads to corruption and a loss of messages. Shriner has found a reproducible bug here and reported it to Apple.
The IMAP and exchange problems don't have a clear source or explanation, and I wonder if they are even caused directly by Catalina. I know I've had Exchange problems in the past and they were almost always caused by a conflict between the huge variety of Exchange servers out there (including very old versions), and the interaction with them by the mail client.
Still, I'm going to watch my IMAP and Exchange folders carefully. I routinely move messages from an Exchange Inbox to a series of IMAP folders, so if the moving message bug is alive, I may still be exposed to it.
With the update to 10.15.2 in Apple Mail I lost about 400.000 mails, which are still displayed with subject, sender and recipient but have no content anymore. Both in the preview and after opening the messages are "empty".
I noticed the whole thing after certain messages, which I normally access via search, were no longer displayed.
I then re-indexed the mailboxes via Onyx without success. Even more messages seem to have completely disappeared from the mailbox folders.
A recovery of the Library/Mail/V7 folder from a TimeMachine backup from 2 weeks ago was unsuccessful because 400,000 elements from the backup were "copied" back into the folder and Apple Mail subsequently indexed the mailboxes (also about 400,000 messages), but these are still all empty in the mail.
Does anyone also have this problem and a solution?
@Patzi. If you have a successful back up of your "on my mac" mail -- restore it to a different location (like your desktop). You should then be able to successful *import* the mail (File --> Import Mailboxes).
That works.
But please file a bug with Apple.
Great advice everybody - thank you!!
This is my issue:
I bought a new MacBook Pro 16". I use Apple Mail consisting of an iCloud address and 2 gmail accounts. I have close to 1.5 million e-mails. I had issues migrating from my older MacBook Pro 15" to this new 16". The older 15" used Mojave, the new one, Catalina. About 1/3 of my e-mails did not migrate to the new computer and many that did, only show headers - no body. Apple customer service really has no good answers. One idea was to upgrade the 15" to Catalina, then try to migrate. But I did try that before and after upgrading to Catalina, Mail froze during the set-up (kept on trying to download e-mail). It was fixed by reindexing and going back to Mojave, so I never got far enough to try a migration from a Catalina 15" to the Catalina 16".
Do you recommend reverting to Mojave on my 16" or trying some way to import the e-mails from the 15" via Time Capsule, external HD, a 3rd party box import program, etc.?
Thank you for bringing this critical information to the forum.
This really sucks. In my case I could still reproduce the problem under Catalina 10.15.2. Every time the reindixing is done more than half of my messages disappear or are only available without body.
@Shriner: thanks for the recommendation. I decided to wait with reimporting until this problem has been fixed by Apple.
Missing mail after Catalina update. Nearly 90,000 historic emails were lost, in my case, reducing the Mail folder size from 35.5GB to 2.7GB. Fortunately I have found an old backup with much of the historic messages. However, key messages remain missing including copies of customer invoices sent out by email. Destroying people's email archives has the potential to put companies, especially smaller folk, out of business. The problem is only obvious once one starts actively looking for those old messages. But, by then, it may be too late ... old backups have been replaced by new backups that lack the older messages. Even if one can recover messages from a backup, the time and effort ( and $$ ) involved in doing so is significant. Shame on you Apple. Protecting user's files/data has to be the #1 objective in any OS update.
Again -- if you have not *personally filed a bug report* -- you are not going to get this problem fixed any faster.
Apple needs to get more bug reports in on this ASAP.
If you use Time Machine you can recover your lost mail. Apple replaced the V6 folder with a V7 folder in Library/Mail. I have no idea how they screwed it up. Doesn't really matter at this point. We just want our mail back.
Open Time Machine and look in Library/Mail for the last backup with the V6 file. Restore it. This will put a copy in your current Library/Mail folder.
Take it out and put it on your desktop. Open Mail and select menu item "File/Import Mailboxes". Choose the V6 file on the desktop. This will put all of your mail up to that date in a folder called "import" in the Mail sidebar. You can drag and drop to selectively replace your lost mail. A bit tedious but better than losing everything.
Happened to me too. Drag and drop was losing attachments. I tried rebuild. Then it said it had to rebuild the index and import everything. 45 minutes later, my emails are massively corrupted. Lots with header only, no body. Many missing. Bunch of random stuff in "Recovered", mostly stuff that I had deleted. Worst problem with my Mac that I can remember in 20 years. Apple, please get on the ball!!
@Sherman — it was interesting to see your reference to FastMail, since that's what I use too. I'm actually delaying purchasing a new Mac because of this, since I'm doing alright on Mojave on my existing Mac, but I'm sure the new one will force a Catalina move. Since I only have FastMail and iCloud email, I'm thinking to just deconfigure iCloud email and go to a single IMAP account, which it appears from this thread shouldn't be a problem running on a single mailbox. I'm also thinking that if I really want to keep both active, maybe I should move to Outlook 2019 or 365, or to Thunderbird, or the FastMail web interface (which isn't great but is good enough), and get out of this quicksand entirely (since all I need to do at most is re-cache my email). It's insane that this is still an active problem by 10.15.3b1.
Just opened a "feedback" on the IMAP-move-to-other-account-issue. Feedback-Assistent lists "no similar feedback" :(
Will reproduce the on-my-mac-issue soon an open a feedback for this too.
I updated from Mojave to Catalina beta a while back and my mail was fine.
When I got a new Mac and reinstalled from a TM Backup via recovery the mailboxes where reimported and that is when I lost about 10.000 mail that were in a local mailbox.
After trying many things I installed a TM backup from before Catalina and grabbed the mailbox file of the local mailbox from a TM backup from the old mac running on Catalina.
I added this to the library of my reinstalled Mojave, deleted the envelope files and after reimport in Mojave I got my mails back from my TM backup that was run unter Catalina.
I will strick to Mojave until this is sorted out.
Best,
Although my issue has nothing to do with Catalina, it does touch on the Time Machine
Mailbox recovery strategies mentioned in this thread. My issue was that I deleted a good
number of emails from my IMAP Inbox before archiving them, and then had to recover them
from Time Machine. At the time of the backup, I was running High Sierra on my machine Rockweed.
Now I am running Mojave. (I have learned to be very conservative in moving to new versions of OSX)
- recover V5 folder from Icarus, which contains my mail from last august:
/Volumes/Icarus/Backups.backupdb/Rockweed/2019-08-26-004934/Saltmarsh/Users/kellogg/Library/Mail/V5
copy to cernboxKellogg/Mail_recovery_2019-aug (note that files in cernboxKellogg are mirrored via the cloud
and available on all my machines)
- with Mail, Import to Import-2 on Rockweed as Apple Mail (since import as mbox claims it can find no mbox)
but when I try to move messages to another local mailbox (2017), I fail, with crazy message that it could not find
Mailbox "Happy New Year from Amherst", which is the title of one of the emails I am trying to move
notice that the mailbox structure is complex, and also that this backup was made under High Sierra, not Mojave
- try the import on maccmskel4, which is running Sierra
for INBOX-mac.mbox I can import it without problems, and I can move all the mails without attachments to 2017
I can view and download the attachments, but cannot move them, getting the warning that the body cannot be loaded
For INBOX-cern.mbox the behavior is exactly as on rockweed: I cannot move any email after importing it
getting the warning that the body cannot be loaded
rebuilding INBOX-cern does not help
- conclusion at this point: OSX versions are not the primary issue (even tried on ElCapitan with same results)
- EagleFiler import succeeds with copy on cernboxKellogg: cernboxKellogg/Mail_recovery_2019-aug/INBOX-cern.mbox (18300 records)
in the EagleFiler window, I see the emails listed very much like in Mail
the # column gives the number of attachments - if I click on the attachment, the entire message (including attachments)
opens in Apple Mail
- Since the Eagle_Library/Files/INBOX-cern flat file obviously contains all the attachments, in-line,
try importing it using Mail/Import as mBox to Import-4
get 18175 messages and a warning that some could not be imported - there are no attachments
Now I get somewhere:
- rename Import-2/INBOX-cern.mbox to 2017-cern and then drag 2017-cern to top level of On my Mac Mailboxes - success!
all the attachments remain visible (this procedure works on all my machines)
- I now delete all the non-2017 mails, and repeat the entire procedure for INBOX-mac and for 2018 and 2019
- only slight issue remaining is that I am not able to find a way of merging the -cern and -mac folders
Conclusions:
- something poisons the structure of the Mailboxes recovered from Time Machine
in such a way that neither EagleFiler nor Mail can handle them properly,
particularly their attachments
- the mail and attachments are recoverable, but their Mailboxes are extremely fragile
Basically: look, but don't touch
@Richard From what you’ve written above, it sounds like EagleFiler handled the attachments properly. But feel free to contact me if you want to discuss further. It’s possible that Time Machine made an accurate backup but that not all of the attachments were cached locally, so they weren’t there to back up. You can avoid that problem in the future by having a rule that moves or copies the messages to an “On My Mac” mailbox or by periodically archiving the messages to EagleFiler.
So, I posted a note to apple communities asking 1. if Apple had acknowledged the bug, and 2. if not, how would I know when it is fixed?
I got this masterpiece of stupidity in response: Apple will neither acknowledge or discuss fixing a bug until it is fixed (assuming there is a bug.)
Sigh.
I've been fighting with this problem for a while and just did an interesting experiment. I sent myself two messages with attachments. One I downloaded to a local folder by dragging it to that folder. The message and attachment survived. The second I moved to the local folder via the Message - Move to drop-down menu. That message only retained its header (no contents or attachment). Has this been observed by others, and does it point to the problem?
Thanks.
OK. I would love to hear if what Doug did on Oct 21 fixed his issues. I updated to Catalina on Nov 30, by Dec 2, I was instructed by the mail client that my mailboxes had issues and before I could finish reading the little pop up window mail crashed. On relaunch mail rebuilt my mail boxes and deleted random emails. A lot of sent emails and all current emails between November 27 and Nov 30. The current ones were obvious, but the others only came to light when I was trying to recover some procedures I had emailed to someone back in May 2019. Those were just gone. After discovering that I started digging aound on the internet and on my machine. Discovered elmx files on timemachine existed, but completely gone on my computer. Called Apple and got no where fast. Spent that last two weekends rebuilding a machine with Mojave and migrating data between it and my current Catalina install... I have recovered from TimeMachine directly to the mojave install and that fails. I have restored the mail and my Library files to an external hard drive for safe keeping. Each time I recover my email from that hard drive and try to put it onto the mojave install the same emails disappear when Mail is relaunched. I am convinced it is because the mail server is an Exchange server, but the sent emails were sent from my computer mail client, not the server... it really doesn't make sense to me that the emails are just deleted off my hard drive by the mail client. I have the email in timemachine and on my external drive, but have been unsuccessful in getting them loaded back into Catalina or Mojave mail. Apple hasn't been able to offer a solution so I am curious if what Doug did with the manual import actually worked.
Something good just happened, but I have no idea why or if happened for others. A massive number (all?) of my formerly blank emails I had downloaded to my computer from my campus server now have their attachments back! The other good news, which makes me think this state of affairs may last, is that when I delete a message from the server using my computer it also deletes from what my iPhone sees on the server. I wonder if there may have been a fix at the Exchange side of thing. Anyway, I'm happy for now and will (cautiously) downloads the many messages I've been leaving on the server for safekeeping.
Cheers.
Where was this obvious issue before last Wed. Went from 15.1 to 15.2 and all of my saved folders with emails for various reasons. On top of this the conversion, it was totally screwed up the interfaces with the provider.
Went to apple support, the first two efforts were absolutely worthless [ very un apple support]. Hours spent bouncing between Provider and Apple. Apple wanted a reboot on Time Machine. That was and still is a mixed bag of bad issues. I will never let anyone lower than tier 2 to go though a time machine reboot.
Finally got a senior adviser. After 45 mins he said this needed engineering. He ran a number of screen shots and zip programs. we short of got thing working only going forward. It is still not stable. My guess we will have to back before 15.2 and go from to square one.
What really bothers me the most, Apple knows there is an issue and refuses to acknowledge it. Then have me running all over the place, causing more issues.
Hello,
Same problem here with Catalina 10.15.2.
Unaware of all these problem with Apple Mail, and because I stupidly erased my mails (pop) while trying to change Pref. Syst./Internet Accounts (that's already strange: deleting an account will delete all related pop messages in Mail…), I decided to used my last Time Machine Backup.
Not working.
Ok, nothing to fear, I restore completely my MacBook Pro 16" with the last backup. Everything is working fine, but Apple Mail: wants to import messages, ok, and that's done, all messages are blank, just headers (and the lthe corresponding folder in Preferences/Mail is shrunk to 5% of its initial size: 54 GB to 2.5 GB...).
Not Happy.
that's already strange: deleting an account will delete all related pop messages in Mail
Yeah, that is so bad UX and it's been that way in Mail forever. Wonder how many out there have been bitten by this? There are so many more reasons to delete an account that just wanting to delete every message stored. Maybe you switch e-mail provider and just want to keep everything as an archive.
I can't talk about any public beta two of 10.15.3 to see if it fixes the issue with deleting the Envelope Index files/reimporting/reindexing "On My Mac" mail.
I would *like* to tell you to try that when it comes out to see if it fixes this problem on your end.
I would like to tell you it fixed my problems.
I really would.
All I can tell you -- it's a good day today. That I'm sure about.
@Shriner, can you not tell us whether or not this fix in the 10.15.3 beta simply repairs the index problem, or does it actually restore the lost e-mails?
I presume that it is not the latter, since, as others have noted, there is more free disk space after the Catalina upgrade, suggesting the actual deletion of e-mails and attachments, and not just an indexing issue.
I lost thousands of e-mails, with Gbs of additional free space after the upgrade.
I am anticipating having to go back to a TM backup from just prior to the Catalina upgrade, and importing e-mails, folder by folder, into the plethora of folder trees that I have, and then dealing with the duplication of e-mails between those imported, and the subset of e-mails that have, so far, survived the upgrade.
BTW, I did file *two* bug reports with Apple on this, since October.
Thank you for not telling us about this...
@Marc -- it will not restore e-mails if Mail.app went through a process (for whatever reason) of reindexing your existing "On My Mac" mail. Those messages are gone for good and would need to be restored from a backup if that happened.
If you have a backup, you should be able to find the backed-up "On My Mac" folder in ~/Library/Mail/V6 and restore that and then "import mailboxes" successfully.
@Shriner, thanks for the clarification/confirmation. It is as I expected, given the observations of increased free disk space post upgrade.
All of the e-mails that I lost were in On My Mac folders. I generally do not keep e-mails in the cloud, with the exception of a few folders for portable and timely access on my iPhone when away from my MBP.
As a second question, I have noted some AppleScript tools online that purport to identify and delete duplicate e-mails. For example:
http://jollyroger.kicks-ass.org/software/
https://www.macupdate.com/app/mac/53180/remove-duplicate-messages
Any recommendations for such a tool based upon any experience?
I would guess that such a tool will be helpful to others in similar situations.
I will await the release of the 10.15.3 stable update and see what Apple has to finally say about this, before proceeding. This process of importing from TM and removing duplicate e-mails will be tedious and time consuming, thus only want to do it once. :-)
Thanks!
@Shriner, thanks for the weather report, very happy to hear about it being a good day as I'm about to have to commit to Catalina, and I just couldn't pull the trigger on moving to another client yet.
Can you also (not) tell us about any (alleged) resolution of the problem with moving messages between mailboxes? Is it a good day today as well?
@dpz -- I only personally move messages from my Gmail account to "on my mac" mailbox locations (that one way -- not the reverse).
So I'm not sure what that problem is directly nor what I'd be looking for to test against.
I would recommend watching for the public beta of 10.15.3 b2 and testing it?
@Marc -- no experience with "duplicate e-mail detectors" here, either. But, what I'd probably say would conceptually work would be that if you know the date you upgraded to Catalina, you could restore your "on my mac" mail from right before that upgrade, import it (which still works fine), and then just purge mail from the leftover "On My Mac" mailboxes from that date and before? That's what I did here. No duplicates remain doing that.
I am seeing some Exchange-related misbehaviour: all of a sudden, about 48 hours, my Catalina iMac stopped being able to fetch or synchronise email with my institute's Exchange server. New messages appear if I rebuild the mail boxes.
It was working absolutely fine before.
This occurs on both a wired iMac at my office and my home MBP on broadband. iOS mail and the Outlook web client work fine.
Moving lots of email out of the Inbox (from about 10k down to 1.7k) doesn't help.
Other accounts (Gmail, icloud) work fine on all machines.
This does not appear to be Catalina specific, however. Refs:
https://apple.stackexchange.com/questions/350372/apple-mail-always-need
https://discussions.apple.com/thread/8086520
https://discussions.apple.com/thread/8152487
Is it, nonetheless, related? Any ideas?
Hi, I just tested the actual Dev-Beta (10.15.3 beta 2) with Mail 13.0 (3608.60.0.2.2) and I can no longer reproduce the mail-loss after rebuilding the envelope-index.
Can someone else confirm this?
I guess i was both sad and happy to find this article. I removed my envelope index files and one of my inbox lost about 110,000 emails. I have a super duper backup where the emails are intact which i can boot to, however i cant export the mailbox from apple mail it just crashes.
Can i replace my broken mail setup with the version from the super duper backup. However i have looked and i cant figure out how to do this. Can i just copy the V7 folder in its entirety ?
Secondly i did actually keep my envelope files before the issue. Can i restore them to fix the it ?
Also like another user here i actually deleted all my pop3 emails from my server as a way to debug the issue :(
Any help would be great. Alex
@Mathew -- yes, it seems fixed here as well.
@Alex -- what you should be able to do from your backup would be to find the "On My Mac" mail folders on your backup drive and "File --> Import Mailboxes" from Mail. The act of *manually importing them* -- is fine. Just don't delete the Envelope Index files as a method to "reimport" them until 10.15.3 is out
Apple fixing it or not, having lost and hence having had to restore from backup 200k mails, I no longer trust Apple worth my mails. I’ve moved away from Apple Mail. My company uses outlook anyway. I’ve set up DEVONthink Pro so that it imports my mails from Outlook. All my mail act on rules I’ve implemented in DEVONthink and I’ve linked them all together with a simple Apple script that I call from my DEVONthink toolbar. Effectively, I’ve now a very simple Auto File procedure that takes my Mails out of outlook (or Apple Mail, but as I said, I’m not using it anymore) and into my folder structure in DEVONthink.
Search and retrieval is anyway far superior in DEVONthink, so there’s no point going back even if Apple finally fixed it. They never really acknowledged the problem in the first place, hence why should I trust them.
@shriner thanks for the advice so how do i select specific mailboxes using this method or do i have to basically import the whole V7 Folder and deal with the duplicates ?
@emax i too was thinking of going down the devonthink route to have a backup system aswell. Though other options i found that had decent reviews were
https://www.mothsoftware.com/index.php
https://mailsteward.com/
So I updated to Catalina ... had loads of problems and had to re-format fusion drive - a common problem apparently.
I re-insatalled from time machine
I could see about 100,000 inbox emails in my mail - but they have no content. About the same number in Sent emails.
After many conversations and trying a few things with Apple - been advised that the only way is to import the emails from the Library / Mail folder for Time Machine manually - which I have now done.
I now have double the amount of emails - the naff ones that can't be read in the main Inbox at the top ... and the same emails that can be read in (Import under 'On My Mac')
I have discussed this further with Apple and they have insisted it is not an apple problem - but I want to be able to get the emails for the bottom import folder into the correct Inbox folders at the top .... as searching emails is really difficult - having to dig down into folders and searching in each folder.
To make things worse I have about 6 or 7 different business emails which each have their own email folder in the top section and now in the bottom section .... But if they are run the correct inbox at the top - you can easily search all at once.
Any advice would be greatly appreciated...
Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed.
Will mail 13 be also available under Mojave when it comes out?
I cannot go up to Catalina and my email suffers from 'eAlzheimer' since upgrade from High Sierra and despite multiple envelope rebuilds
thanks for your feedback
RE >>> Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed. >>>
is there an estimated time for this ?
as mentioned above I have all emails - but they are all in an import folder below - not in my inbox - which makes searching for emails a pain.
I may try and drag them all over to inbox - but if there is a fix - is it worth waiting ?
So -- for the purposes of the bug that I know is fixed -- the problem was with "On My Mac" mail being deleted if "Envelope Index" files were deleted (or Mail.app threw up the dialog that the Mail index needed to be rebuilt from the first launch of Mail after upgrading to Catalina). This is fixed in the current 10.15.3 betas. I'm confident in stating that *specific* bug -- seems fixed.
I've not heard (here) about a way to replicate mail being deleted when "moved between mailboxes". I'm happy to try to replicate it on my end with some test accounts and open a case, but I'd need more detail like -- what kind of mailboxes -- and in what direction -- is the mail being moved? Exchange? Gmail? Moving between "IMAP" mailboxes? Pulling messages down from an "Inbox" to an "On my mac" mailbox? Moving message *back up* to an IMAP mailbox from an "On My Mac" mailbox? Is this all POP (for which I'm surprised people are still using that protocol these days...)? Exactly what is the setup where you see that specific problem?
To answer a couple of specific questions above (and I don't work for Apple -- just a sys admin at a large University so I see a lot of stuff...)
Mail is not going to be updated in Mojave (old OSes only get security fixes once a new OS comes out -- Applications are never majorly updated on N-1 or N-2 OS versions.
If you are trying to restore deleted "On My Mac" mail from a TimeMachine backup, I would recommend restoring the entire backed up ~/Library/Mail folder to another folder on your desktop. After you do that, you can look inside the folder for the mail structure. "On My Mac" mail would be within it's own folder with a long string name like: 0D214F03-E8D2-4DAD-8F4F-6285203B4475 or something and you can look inside that to identify the mail folders within. That would be the folder you would "File --> Import Mailboxes" only. No need to reimport *everything* to solve that specific problem (as you'll get duplicates of stuff you don't need.)
And nobody knows when 10.15.3 will be released apart from Apple. When the betas stop coming, you'll know it's close. *In general* (but not always) when beta build numbers end with letters closer to "a" -- then it's close to release. Beta 3 is 19D75a -- so it *may* be closer to release than I would have guessed considering there have been only 3 betas. Could be this week? Could be next week? Could be mid-Feb. Nobody knows.
According to this article:
macOS 10.15.3 is released, albeit, it is not yet showing up for me as an update.
I am not seeing any explicit indications of a resolution of the mail bug in the coverage, at least not yet.
Sorry for the second rapid post here, but the 10.15.3 update is now showing for me, with the following details only:
This update:
• Optimizes gamma handling of low gray levels on Pro Display XDR for SDR workflows when using macOS
• Improves multi-stream video editing performance for HEVC and H.264 encoded 4K video on the 16-inch MacBook Pro (2019)
along with the following security updates:
https://support.apple.com/en-us/HT210919
So, still no official acknowledgement from Apple of the mail issue...
[…] Tsai, who has been tracking this issue, […]
Can anyone here confirm if is possible to take a backed up V7 folder and replace the broken V7 folder in its entirety ?
I understand anything not the old backup would be lost.
Thanks. Alex
@Shriner Here is my experience with loss of messages.
My account is on Gmail / IMAP. I can send all the settings if needed but the most relevant may be that the "Archive Mailbox" is [GMail]/All Mail. I have numerous rules for filing messages. This basic configuration was stable for many years until I upgraded to Catalina.
When a new message comes in, it is just fine no matter what mailbox it has been filed into. If the message was not filed by a rule (i.e. remained in the Inbox), moving the message to a different folder works fine. But if the message was already filed by a rule, as soon as it is moved to a different mailbox, the content is lost. Mail displays the sender and subject, but nothing else. "View Raw Source" displays an empty window.
I imported all emails in 10.5.3, but moving emails to a different folder in the "On my Mac" is only partially working, some messages move fine, while others only show empty content. Rebuilding the mailbox make them disappear completely. I've explored the V6 folder, and it seems that all "xxxxxx.emlx" files are imported/moved fine, but the "xxxxxx.partial.emlx" files are sometimes not imported and are the ones that disappear when moved from one folder to another. What's the difference between the emlx and the partial.emlx? Is this what is happening to other users?
Hello everyone.
Since Apple doesn't seem to fix the problem, I have two questions and it would be wonderful if anyone could help:
1. Does Spark Mail have the any issue at all?
2. How can I migrate to Spark. I wonder if I cancel my Mail® accounts it will delete / free the space occupied by the mail messages 🤔...?
Thanks in advance,
Problem with mail.
With MAIL in the dock, I used to click on the icon once and it would open. Ever since I upgraded to Catalina and then the latest version I have to double+click the icon to open MAIL. Anyone else have this problem? I talked to Apple and the advisor said they were aware of this problem.
OK,,,,,, Fix it Apple!
@michael tsai — Regarding your reply to @Alex.
@shriner — Regarding your request for details.
My experience is as follows. My V7 was created on a brand new 16” MBPro running 10.5.2, by using the 16” ’s Migration Assistant feature and using Time Machine containing my V4 (Sierra) as the Source. The resulting V7 did not contain the same number of .emlx files contained in the V4. When I initiated the 16” Catalina Mail App, I was missing text of the emails in both the upper level “Inbox” Mailbox (POP3) and the lower level On My Mac individually named Mailboxes.
@michaeltsai—Thus, I believe the V7 folder is not reliably created from the prior version V# folder, and any future “backed up” V7 folder will itself be broken. Thus, @Alex will not be able to use a backed up v7 folder (which I believe will be broken itself) to fix his current broken v7 folder.
@shriner— do you have any experience similar to mine?
Hi All,
In light of the last few comments, I want to verify that there appears to be lingering issues with Apple mail subsequent to the 10.15.3 update.
That is, not all data loss related Apple Mail bugs/issues that have been observed and reported are resolved by this update, especially those with the "On My Mac" local storage folders, which is where the vast majority of my mail is stored. I keep only current Inbox messages or those requiring expedient attention in a "cloud" based IMAP Inbox folder tree, whether that be a personal me.com account, or my business account, which is via Office 365.
If correct, which would be concerning, I am hesitant to begin the tedious and lengthy process of restoring hundreds of folders/sub-folders from a pre-Catalina TM backup, after the 10.15.3 update, which I have installed.
@Mathew's comment above might open the door to suggest that Apple did not ship the beta3 version of Apple Mail in the 10.15.3 release, which might perhaps explain lingering issues, and that perhaps there will be a future 10.15.4 update to Catalina that may yet resolve other mail issues.
I am also, for the first time in years, contemplating moving to a different e-mail client application on both macOS and iOS, given these issues, especially if they are not fully resolved, and there has been no ownership by Apple of them.
I already have an Office 365 account for other business related reasons, as noted above, so moving to Outlook as a macOS/iOS client for both personal and business accounts would be one pathway, albeit, I have seen data loss reports in past years regarding Outlook. I would, of course, still need to restore the pre-Catalina e-mails from TM, but only really want to do that once, whichever pathway relative to an e-mail client I take.
Any comments would be appreciated.
@Dave SpamSieve can use any mailbox that you want. Most SpamSieve users are already on 10.15, and for 99+% of them there are no issues. If there is an issue with your setup, it should be immediately obvious when you test it out with a copy of a message, e.g. you drag and drop a message and it disappears or Mail rules fail to move messages. In that case, some people have found that Catalina Mail is better at moving messages to a mailbox on the same server as the inbox rather than to another account or a local mailbox. Some other workarounds are described here . SpamSieve can also be set not to move messages when training, if you happen to be one of the people affected by the move-messages bug.
@Michael, thank for your comments.
Unfortunately, I cannot be assured that messages in the local On My Mac folder tree would not have to be moved from one folder to another. Thus, raising integrity concerns, if that is still an issue.
So, it would seem that one path is waiting for some indication from folks here, that perhaps an upcoming 10.15.4 beta release does make progress on resolving these issues, or does not.
The alternative would be, if either the above does not show signs of resolution, or Apple simply takes too long to resolve this (for some definition of "too long"), to consider moving to a different client application as noted.
@Michael, perhaps I misunderstood your prior comment to me regarding having e-mails in the On My Mac folders and not moving them, in which case they would be ok. The logical inference would be that moving e-mails between folders might not be ok, albeit, perhaps that is not confirmed, per your follow up comment.
If there is any reasonable question, at this point, in light of the underlying issues, of lingering data loss, I am not prepared to begin the process of restoring/importing the old TM backup e-mails. As they say, the only good thing about banging your head against the wall, is that it feels so good when you stop.
I can't be selective in which operations may or may not be safe from a data integrity perspective, and honestly, shame on Apple for engendering those concerns and doubts.
@Marc Moving between IMAP and On My Mac, I have reports of problems. Moving within On My Mac, I have not, but that doesn’t mean that there are none. As I wrote to Dave, it seems like some people are affected (frequent problems) and others are not (none), and so if you figure out that you’re in the latter group you’re probably OK. But this is all guesswork because Apple has the code and bug reports and isn’t talking (even about the bug that seems to be fixed).
My general practice is that *everything* gets moved to an IMAP server once it arrives—most incoming mail comes in through an Exchange server at work, and I manually folder it to a different IMAP server after I've reviewed it... so moving between an Exchange inbox and a different set of IMAP folders.
From reading the reports above it wasn't clear it me if others have reported problems with this style of setup; most seem to concern "On My Mac" foldering.
Is anyone using a similar setup to me, and have you experienced problems with Catalina?
@John, for clarification, are the rules that you are referring to, rules that you created in Apple Mail, or rules created on the gmail web site?
I ask because I have a number of rules on iCloud.com, that move me.com mails to IMAP sub-folders of the Inbox for e-mails lists or similar differentiation for organization. I did this on iCloud.com so that I get the same behavior on my iPhone as I do on my Mac, relative to folder movement.
I only have a few rules in Apple Mail, that based upon keywords or similar, color the subject line, but they do not move e-mails to a different folder. I don't see this, of course, on my iPhone.
@John
I have filed this bug with Apple in November : FB7449043. Have no response to it whatsover (hasn't even been marked as a dupe so it appears that it has gone in to their ignore bin despite the DATALOSS in the subject line.
Here is basically the text of the bug:
DATALOSS: Messages automatically moved by rule on receipt lose data on subsequent moves
Mail Version 13.0 (3601.0.10). On macOS 10.15.1 (19B88)
When a message is received and automatically moved by a message rule to a local (On My Mac) mailbox, subsequent moves of the message (by drag and drop or by menu move options) result in the total loss of message data.
Steps to reproduce:
1. Create a rule to move a message to a local mailbox (Mailbox 1)
2. Receive a message and confirm that message has been automatically moved to the local mailbox (Mailbox 1) and has its contents (headers/body etc)
3. Move the message (via drag and drop or the menu move actions) to different mailbox (Mailbox 2)
4. View the message in Mailbox 2.
Expected result: The message in Mailbox 2 has it headers and body.
Actual result: The message in Mailbox 2 is completely devoid on content -- viewing raw source shows no data at all.
Note:
• Reproducible 100% of time
• No plugins are installed/active.
• Occurs regardless of type of receiving server (IMAP, EWS, Gmail,iCloud).
• Occurs also if the rule copies (rather than moves) the message.
• Only occurs if destination mailbox of rule (ie Mailbox 1) is local -- if Mailbox 1 is on a server, subsequent moves are OK.
• Mailbox 2 can be any mailbox -- (ie may be local or on a a server).
• Messages received to the inbox and subsequently filed by manual application of rules are NOT affected.
• Messages received to the inbox and subsequently moved by manual moves of message are NOT affected.
I did more research at the time and determined that the message moved by the rule was lacking a specific bit setting in the message flags when compared to messages moved manually.
If I manually edited the sqlite database for Mail to turn on this bit for a message. (as a plugin developer I do a lot of spelunking), the message was moved without dataloss. So in my estimation, the rule application is not properly setting a specific bit on the message that needs to be set.
I communicated this observation in my bug report -- which if correct should lead to a quick fix. But that was 2 1/2 months ago.
@Scott Thanks for this information! I think this is just one aspect of the problem, though. Some users are seeing messages disappear when a rule moves them from the inbox (on a server) to another server mailbox. And I have also seen reports of messages deleted when dragged and dropped between mailboxes on two different servers. I have not been able to reproduce these myself, though.
Update (2020-02-18): I have received a report from a customer with macOS 10.15.3 that seems to indicate your bug is not fully fixed, either.
I'm currently running Mojave and very reluctant to upgrade to Catalina because of the mail issues. However, I have installed Catalina on a VMware Fusion VM to do some testing and have found no issues with my Fastmail IMAP account. All my rules are server side so emails get moved into the appropriate folder at Fastmail's end. I don't use local mailboxes on my Mac - just the ones in my Fastmail account. I've done some manual moves of a large selection of emails back and forth between a number of different mailboxes and all seems OK.
Just to confirm - this is a brand new install of Catalina setting email up from scratch, not an upgrade from a previous version of MacOS.
@Scott, I am pleased to see that you filed such a detailed report to Apple, and that you have observed a possible fix in 10.15.3.
I will update this weekend and confirm whether I also see the issue has been fixed.
I just read the blog and im not sure whether this is related to my issue but whenever i move some of my mail to its relavant folder since im using the mail app, i lose all its content and all i have is its subject title. Can you halp solvong this because i dont understand how it started and how to stop it.
I have a friend who is currently using MacOS Mojave 10.14.6 She has IMAP mailboxes, but also quite a bit of older messages stored in folders in On My Mac. Not using any rules, and not very likely to be moving messages between On My Mac folders. No longer has any POP3 accounts. Is an update (not clean install) 10.15.3 release still a risky proposition, simply because she has a lot of old messages in On My Mac? It looks from what I see here that Apple has made some progress, but there are still some remaining issues, especially with moving messages in On My Mac.
Here are two possible solutions to the problems we are all having, which I've copied from the Apple discussion forums. If anybody's brave enough to try them, I'd be mightily interested to know the outcome:
I have solved the Catalina email loss problem and solved the email indexing problem! YEAHHHHHH!
I had over 500,000 emails, with more than 30 POP accounts and 15 IMAP accounts. I had over 200 folders and sub-folders. A disaster. I had updated from Mojave to Catalina and I realized the problem when I went to do a search without results. I had tried them all, but the emails disappeared. I contacted Apple's specialist support, but they didn't know how to answer, although many users complained about the problem. I will illustrate the solution;
1) obviously I had a manual backup of the Mac library on Mojave. The important thing is to have a backup where the emails previously worked. regardless of whether catalina or mojave. In catalina, only the name of the V6 changes to V7 but 99% is the same. So you can recover and run everything, even from a mojave backup to catalina
2) I restarted the Mac in safe mode (press shift button)
3) I entered the mac library and replaced all the contents of the ACCOUNT folder (library> account) with the content saved in the backup
4) I entered in Preferences (library> account) and I replaced these two files with those saved in the same position in backup hard disk:
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis
5) I went into SyncedPreferences (Library> SyncedPreferences) and I replaced these two files (if there are none it does nothing) with those saved in the backup:
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist
6) I went into Containers (Library> Containers) and replaced the folder saved in the backup:
com.apple.mail
7) Now go into Library> Mail> V7 and delete everything. From the backup disk, first I copy and paste the Maildata folder (before Catalina automatically rebuilds it - it is the first folder to copy) then I copy and paste from the backup disk, all the other folders in blocks of 10 at a time (so that if Catalina blocks me saying "this folder / files already exists, I can start over from the last point).
8) When I have finished replicating all of the V7's internal from the external hard drive to Catalina, I restart the mac in recovery mode (Option-Command-R or Shift-Option-Command-R to start up from macOS Recovery over the Internet ) and do an SOS on both Macintosh HD discs. Then reboot
9) Now with the active connection, I open MAIL and wait for all emails to be indexed. When Mail re-opens all the e-mails and folder trees, I will have a time of about 30 -40 minutes, where all the e-mails, the body of the e-mails can be seen correctly before catalina deletes everything. The first thing I do is to scroll (even quickly) all the folders to the end, so that the indexing of Core-Spotlight can load the contents-preview messages.
10) Attention! At this stage, indexing and email search still does not work. Whatever you are looking for, it still does not work.
11) I make sure there are no folders with no body messages, so I click on each folder, sub-folder, sent mail, individual sent mail folders, one by one, (I did it for 200 folders, pop, imap, pec , sent, archived, icloud etc) and click on> MAIL> BOX> REBUILD (be careful you have to do it one by one and wait for it to finish rebuilding).
12) I suggest storing the number of messages (eg 14.234) so that when it rebuilds, it returns 14,234 messages.
13) Do it folder by folder and you will see how everything will stabilize and also spotlight and core-spotlight, it will give you the results.
If you have solved it, send me a like on my linkedin profile: https://it.linkedin.com/in/massi-vigilante-741837147
I summarize:
1) boot safe mode and connect disk backup
2) replace the folder library> accounts, the files in preferences, the files in syncedpreferences, the files in containers from the hard from the external hard disk in users library
3) delete the contents of the Mail> V7 and starting from the Maildata folder, copy the contents of the v7 from the backup to the internal library
4) restart in internet recovery mode, do sos
5) open mail after restarting the PC wait for the emails to be indexed
6) scroll all the folders with the mouse to load the content
7) folder after folder, reconstruct (mail> box> reconstruct)
8) wait for it to rebuild and then move to the next folder
Reply Helpful (1) Thread reply - more options
Several people have reported lost or blank e-mail messages after updating macOS to Catalina. I lost 50,000 messages. Here's my solution, which requires a back-up of messages made before the Catalina update.
Restore the folder V6 from User/Library/Mail to the desktop.
Open Mail if it is not already open.
Create a new mailbox where you will put recovered messages so that you can review them before storing them with other messages.
Select the command "File" - "Import Mailboxes … ", then select "Apple Mail" as the format.
Look in the folder V6 on your desktop for a mailbox that you want to restore, select it, then click "Continue". The import will start. A message will tell you when the import is done.
After the messages have been imported, the messages will be in a new mailbox called "Import". Within that folder, they will be either in the simple folder structure that you would expect or in a multi-level folder hierarchy. If they are in a simple folder structure, click on a message in the message list, press command-A to select all the messages, and drag them to the folder that you created in step 3. If they are in a multi-level hierarchy, option-click on the triangle next to the folder called "Import" to expand the hierarchy. Command-click on all folders named "Messages". Click on one of the messages in the message list. Press command-A to select all messages in the message list. Click and drag the messages to the new mailbox that you created in step 3.
Progress in moving the messages will be shown at the bottom of the folder list. After all the messages have been moved, delete the "Import" folder by right-clicking on it and selecting "Delete Mailbox … "
Repeat steps 3 to 7 for other mailboxes.
Methods that did not work for me:
Re-building mailboxes.
Deleting envelope files in folder V7 and forcing Mail to re-index V7 messages.
Temporarily removing folder V7 and forcing Mail to re-convert messages from folder V6.
Restoring a Time Machine back-up from within Mail. Mail crashes.
So it looks like the beta cycle for 10.15.4 has started:
Just sayin... ;-)
Boy there is a big jump. I just installed the new macOS-Beta (10.15.4 beta 1) and Apple Mail is now Version 13.4 (3608.80.13.2.1).
macOS 10.15.3 (stable) has Version is 13.0 (3608.60.0.2.5) so here could come a cure for a lot of bugs.
Sadly I am not able to reproduce any bugs in my VM but at least we can see Apple will ship a greatly improved version of Mail in 10.15.4.
This is the longest time that this thread has been quiet since it started :-)
Any feedback fom macOS 10.15.4 public beta 1?
Maybe relevant for others. My issue was that my local mailboxes where empty after importing from a mail export of my previous machine.
I could finally solve this by importing each local-mail subfolder (first level) separately.
It seems in my case that importing the main local-mailbox did not lead to recursively import subfolders and was only creating them empty.
when importing each one by one their recursive content was imported.
Many, many THANKS to John Biggins for this fantastic information that really helped me. I lost about 75000 emails when upgrading from Mojave (10.14.6) to Catalina (10.15.3). My previous macbook pro had to be taken into the macstore for repair for the 4th time (2 logic boards replaced, the top case was replaced twice and the touch bar had an intermittent connection) and rather than fix my machine, they gave me a new 16” machine, which I love, but I was forced into upgrading to Catalina. :-/
Just wanted to make a few notes to help make this clearer for others that may be going through this. I have noted below according to the steps in the first part of his message.
1) i had a time machine backup of 10.14.6
3) when he mentions ACCOUNTS, he is talking about USER account, so what made more sense to me was this: Macintosh HD > Users > YOUR USER ACCOUNT > Library - you don’t need to replace all of the content from this folder - only what is relevant to Mail
4) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Preferences If you dont’ have these files, don’t worry about it
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis
5) Macintosh HD > Users > YOUR USER ACCOUNT > Library > SyncedPreferences If you dont’ have these files, don’t worry about it
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist
6) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Containersl
7) if you’re coming from Mojave, you will copy from V6 to V7 in Catalina I copied the Maildata folder and then everything else at the same time. took about 10 minutes to copy
8) I wasn’t sure what he meant about SOS, so I just ran First Aid disk repair - on both the Macintosh HD and Macintosh HD - Data disks
9) I didn’t understand what was meant about Catalina “deleting everything” and luckily didn’t run into this.
11) On my machine, nothing really happens when you hit Rebuild - the emails from that folder might blink. on the folders that had thousands of emails, if I clicked on an email, and then another email, and then selected the first one again, the sender would disappear and the body of the email would disappear. this means that it is rebuilding. this will all come back. Also, the rebuild command in the menu may grey out. when it’s ready again, it will appear as selectable again
12) I opened mail on a back up computer and checked folder by folder to confirm that all of the emails were, in fact, there.
Can anyone that is running 10.15.4 beta 2 and 3, care to not comment on the behavior that they are not observing relative to the bugs? :-)
Thanks!
I also today found Mail is only giving me Headers in some emails and on other the full Header and message.
Cant figure what to do I pad and iPhone emails are all ok.
Any ideas anyone.
Today I created a Smart Mailbox to view selected messages that were stored in a mailbox in On My Mac.
I then manually moved the messages from the Smart Mailbox to a new mailbox in On My Mac (and, in case it matters, also created new Rules for that mailbox).
When I opened the messages in the new mailbox, they had lost their content. The headers are there but the body of the messages is blank.
When I exported the new mailbox, then imported it back into Mail, the content returned. (The status of the messages, such as read/unread, replied to, etc., is lost though.)
So, apparently the content is only hidden, and maybe someone can figure out how to display it again without exporting and importing.
Migrated to Catalina and have had the usual problems outlined above. Still have the e-mails on hard drive so imported them, but then found out that I had lost folders so tried Time Machine, but then Mail crashes and cannot use the backup. However, now have most e-mails back, so just crawling around to see what I had lost in the upgrade...one thing I've gained with Catalina, though, is a load of flashy applications that I don't need and don't want...such as 'News' and 'Stocks'...
If I perform a fresh Catalina install and pull down mail from Exchange, am I going to have issues with mail disappearing when they move folders?
I move mail all the time, usually via an AppleScript I wrote years ago. I assume that no matter how mail is moved (by hand, via script, by local/server rules, SpamSieve mark-as-good), the problem exists?
I've turned apple mail to offline. I still use exchange. I'm using ostbox for email and mailsteward to back up (long time user, need to give eaglefiler the time it deserves).
Does this mean it's reasonably safe for me to upgrade to catalina? Still not sure I'm gonna do it, but I am curious.
@Michael, I am seeing a variety of "flaky" behavior still, especially when viewing messages that have been moved between IMAP server folders on my MacBook Pro, in contrast to viewing the same messages on my iPhone, which I also upgraded today.
Synchronization issues abound in the IMAP Inbox and sub-folders that are on the servers. I have had other message content issues with moving mail from IMAP server folders to local On My Mac folders.
This is getting disheartening, in all honesty, given how long it has been going on.
I am about ready to give up on Apple Mail, and migrate to Outlook. I am finding myself, more and more frequently, needing to go back to e-mails from a while ago, that disappeared from Apple Mail local folders during the Catalina upgrade. I have yet to attempt re-importing the old mail from a TM backup, given how time consuming a process that will be, which makes accessing those older e-mails a pain.
Any thoughts from your end that might alter my views?
Thanks for maintaining this post.
@Marc
I concur with Marc, thank you for maintaining this post Michael.
My institution gave up on supporting Mac Mail because of this same issue. I still use Mail, but am spending more time on chasing missing attachments than on my research. Using Outlook and Mail simultaneously is a recipe for madness, not productivity. Are people bailingand moving to Outlook in the short term? ...is that even possible from Mac Mail? ...is there some .mbox > .pst converter out there that actually works?
@Bathead, I have not yet done a lot of research, but here is one approach:
It seems to me that another option, presuming that you have the emails that you want to migrate in local Apple Mail folders, is to move them to IMAP folders on the server for the relevant account, for example, iCloud. This presumes that the emails and any attachments are intact.
Once they are in the IMAP folder on the server, you can then access them in Outlook, and then move them to local folders in Outlook.
Bear in mind, that at least for an iCloud account, you would need to create app specific passwords for security to use third party apps, like Outlook, presuming that you have 2FA enabled.
> Using Outlook and Mail simultaneously is a recipe for madness, not productivity.
For whatever that's worth, I use Mail for Mac, Outlook for Mac, Outlook for Windows (on two separate machines), and Outlook for iOS simultaneously on the same company account. Doesn't seem to cause issues.
While it's gotten better over the years, there are still things about Outlook for Mac that just don't feel right, and also things compared to the Windows version that are missing. For reading, I think I generally prefer Mail (and have been lucky, I guess?); for writing, there are some compatibility issues with the way Mail sends formatted mails, so I avoid that in official communication.
> there are still things about Outlook for Mac that just don't feel right
You mean like having Undo not undo your most recent action but instead randomly pick some other of your past N actions, often so far back in history that you can't see what actually was undone?
@Mathew For some users, dragging and dropping a message from one mailbox to another (e.g. from the inbox to another mailbox on the same account or from an On My Mac mailbox to a server mailbox) makes the message disappear. And the same thing happens for other methods of moving messages (rules, AppleScript). I have multiple reports of this still happening on macOS 10.15.4. Moving to an On My Mac mailbox seems to work better.
I have been wanting to move from MacOS Mojave to Catalina. Part of my e-mail history is a large group of messages that date from when I was using POP e-mail - "On My Mac". I currently use iMAP exclusively, and have I have multiple IMAP folders.
Michael said above: Yes, that should be safe as long as you update to 10.15.3 or later, so that you get the fix for the bug with locally stored messages.
If I update from Mojave 10.14.6 to Catalina 10.15.4 AND I don't attempt to move messages between folders on my Mac, am I likely to still have problems? I currently use an iPad Pro when receiving, sending or moving mail to folders.
@Gordon - note that I haven't read that the problem is moving messages between folders, per se. The problem is moving messages between mailboxes (i.e., accounts). Others should correct me if I'm misinterpreting that.
I'm setting up a VM to play with Postbox and Thunderbird. I'm done with Catalina Mail.
I tried all of the mail programs that @Michael recommended above and have chosen MailMate. The most important thing to me is to have "Smart Mailboxes" that include several email accounts. I also needed a flexible rules engine. Thanks @Michael for your recommendations. I am sad that Apple clearly doesn't consider this a problem. I have used Apple Mail for many years.
FWIW, I just filed what is my third or fourth bug report on this issue.
I would urge others to do the same, so that Apple is aware of this issue and gets feedback from multiple sources.
I included a link to this thread in my report.
I wanted to add a bit of info to this thread. I have not personally updated to Catalina, due solely to this bug—I just can't risk losing old email messages—but I've been monitoring this thread since October to see how things are progressing.
Thursday evening I decided to do something I've never done before, write to Tim Cook: "Tim, I’m sure you and the rest of the Apple senior leadership are slammed dealing with the COVID-19 outbreak. But I’m writing due to an ongoing, and very serious, bug in Catalina that doesn’t seem to be getting engineering attention. Catalina, even as of the 10.15.4 release this week, has a disastrous Apple Mail bug that can result in permanent loss of messages stored in remote servers and locally. Here, for example, is a discussion on a seasoned Apple developer blog about the problems that have been ongoing since October 2019 when Catalina was released: [link to this thread]. I hope you can make sure engineering is aware of the issue and is dealing with it, because it’s a show stopper for many of us."
I was shocked to get a call the very next day from a member of the Apple Corporate Executive Relations team who spoke to me about the problem and promised to look into it with engineering, and get back to me today. Today he called back and let me know that he couldn't share anything specific but indicated that now that the issue had been raised with Tim, Engineering would have to look into it with high priority and report back to Tim about its status. He said they normally would have put me in contact with someone from engineering to work through diagnosing the problem, but since I'm not on Catalina yet that's of course not an option. I made sure he was aware that multiple bug reports had been filed against this issue and he said they would find these as they look into the problem.
So I'm not sure what this means, of course, or whether it'll lead to any action. But I figure it can't hurt, and I was impressed by the speedy and high-touch response I got.
@Keith, wow, that’s great!
I was coming here to post that Apple has released the 10.15.5 betas to devs, and am wrestling with waiting to give one more stable release a try or move to Outlook.
Your post may give me the basis for waiting.
Thanks!
Sitting here looking at the update notification for at least the 100th time, I wondered if there had been any movement on this issue. Quite surprised to find that it's still a thing. I have an IMAP account with hundreds of thousands (million(s)?) of messages as I rarely delete things. It's just too useful and easy to be able to go back in time.
I wouldn't want to lose that. In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.
Glad to hear the issue seems to have been successfully escalated by you, Keith. This needs to get fixed before the next major release. Please keep us up to date - at least some of us are listening.
*Clicks "Remind me tomorrow" again*
> In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.
I think Michael Tsai might have one? (Not sure how open the format is.)
And it's Postbox FTW after ruminating on Postbox vs Thunderbird for about a week. Bye Felicia.
Thanks for the all the guidance here!
Keith,
I also really appreciate your making that effort to get this bug some attention, as I am in the exact same situation that you are. Please do keep us posted!
Frank
Does "display mail as plain text" make blanked emails readable?
Btw: if some here has Tim Cook on the phone: in previous version of OSX you could sort each folder in finder differently and the shared folder was usable, while today (10.14 and 10.15) a new file added to the shared folder is paradoxically not accessible from other profiles on my same iMac.
So I have been testing macOS 10.15.4 for about a week now.
I have one POP3 account an one IMAP account set up in Apple Mail. Both running on a Kerio Connect server.
I can't find any misbehavior so far. Moving messages from POP or IMAP inbox to local (On my mac) mailboxes and between them works. Tried it by drag an drop by mouse, by TouchBar and with mail-rules. No mails lost, no content corrupted. And I have moved quite a bit, way over 100 mails.
So can anyone give more specific input on the open bugs? Maybe it is no longer an Apple Mail bug but for example a Exchange our Outlook-Mail-Server bug?
Kind regards, Mathew
@Mathew The message moving bug seems to only affect a small percentage of users, but for them it is frequent. So if you haven’t noticed any problems, you are probably one of the lucky ones. For the unlucky ones, it affects a variety of servers (iCloud, Gmail, Yahoo, various IMAP, Exchange), so I don’t think it’s due to a server bug.
@Mathew and @Michael, the other point I might make, is that if this was a server bug and not application specific bugs, you would reasonably expect to observe the problematic behavior in other mail applications (e.g. Outlook, Thunderbird, Postbox, etc.) connected to those same problematic servers.
That would include Apple Mail on pre-Catalina versions of macOS.
I don't think that we have seen such reports here, to my recollection. Thus, the common denominator would seem to specifically be Apple Mail on Catalina.
@Mathew
I made a clean install of 10.15.1 and restored my profiles from my TimeMachine Backup (as of 10.14.6). All downloaded emails (pop3 and those manually copied from my IMAP account to my local mailboxes were blanked. On both OS X profiles. I tried two times.
I am to lazy and too pessimistic to try it out once more with 10.15.4. Anyone did?
Apple has released a supplemental update for 10.15.4. Nothing in the release notes about Mail.
https://9to5mac.com/2020/04/08/apple-releases-macos-10-15-4-supplemental-update/
Wanted to confirm that there was no change in the Apple Mail version with the 10.15.4 supplemental update.
It remains at 13.4 (3608.80.23.2.2)
All I know is that I have interminable, unpredictable, never-ending problems with Apple mail, using Mojave 10.14.6 (and iPhone iOS 13.x).
IMAP accounts/gmail in particular seem particularly obstinate to staying connected; continually, repeatedly asking for password, security alerts, etc. particularly on the iMac.
No matter what I do and how many times I respond: YES that was me! and continually re-enter password through safari, update / reconfirm alt or recovery email / phone, it keeps looping back and continually requiring the password to gmail accounts.
I used Howard Oakley's Silent Knight to install recent updates that OS X wouldn't/couldn't despite system prefs being configured to install security updates -- not sure, but I think that's when it started doing this whole repeat asking for passwords in apple mail for all IMAP/gmail accounts.
Really really really frustrating and annoying -- I can't figure out how to fix. What a freeking waste of my time on this repeating, resurfacing bug.
Thanks, if anyone has the fix, please do advise.
@John, my initial read is that the information provided there is at best incomplete, and at worst, misleading and may even make the situation with Mail worse. It suggests a lack of technical understanding by the author.
I say that because several of the methods presented presume that the underlying data files are still present on the local drive. I know from personal experience, and others have posted similar information here, that the mail files and attachments are literally deleted from the local hard drive as a result of the Catalina mail bugs. I literally lost over 30 Gb of e-mail related data, as after the Catalina upgrade, I had 30 Gb of additional free space on my hard drive. If the mail data files were still physically present, and it was just a matter of recovering those files, or rebuilding the indexes to those files, that would not be the case. That reinforces the notion that unless you have a Time Machine or other backup of your Mail data files from just before the Catalina upgrade, you will likely have no way to recover the lost e-mails. Their application will be of no help in that scenario.
In addition, their recommendation for rebuilding the mailboxes may/will actually make the situation worse, resulting in the deletion of additional mails that may have survived the initial Catalina upgrade process.
Bottom line, I would not use it.
So...I have it on good authority that the current build for Apple Mail under 10.15.5 Beta 3 is still version 13.4, build number 3608.80.23.2.2.
No change from the currently released version of Mail included with 10.15.4, with the supplemental update from April 8.
Thus, there should be no expectation of any additional bug fixes once 10.15.5 comes out, unless there is some last minute update to Mail, before it is released.
Not good news...
@Philip, FWIW, if it was me, I would not upgrade yet, presuming that you are using Apple Mail and not another mail client, of which there are numerous options, some of which have been noted in the past here.
I would also be sure that you are diligent with frequent backups, Time Machine and otherwise.
I am still holding out hope that Apple will resolve the remaining Mail issues soon, but if they do not, within the timeline of one or two more incremental updates, I am going to move to Outlook, as I already pay for an Office/MS 365 subscription for business use. It gets increasingly frustrating when I need to go back to past e-mails that have been lost to the upgrade and take time to locate them in a backup from prior to the upgrade. I am not willing to risk restoring those e-mails back to Apple Mail via an import, for fear of losing them yet again.
I have already started to "play around" with Outlook on my MBP and iPhone, just to get used to the pros and cons of making the shift.
If you "need" to upgrade to Catalina in the near future, for some reason, I would give serious consideration to moving to a different mail client before doing so.
if I would remove the synced email I have (eg.g gmail) from mail and then upgrade to catalina and then set it up again (i.e. sync to server) would I then be ok?
If I also have an archived mailfolder, can I make a copy of that before the update and the reimport it to mail in catalina after the update?
If that would work it could be a way forward. Hope for a good answer on this.
Best regards,
Hi Marc,
Am also considering switching over to Outlook. If I download all my emails from the server (going back years) do you think they will survive the update to Catalina?
Thanks,
@John
Nothing is ever guaranteed, but if you have all of your e-mails stored on an IMAP server and keep them there, as opposed to using any local storage, and sync them to a new e-mail client (e.g. Outlook), they should be ok.
If you intend to then move those e-mails from the server to local storage in Outlook, I have not done that yet on my end, but in theory, those should not be affected by the upgrade to Catalina, since the bugs appear to be within Apple Mail in isolation. I won't do that myself until I am fully ready to move to Outlook and abandon Apple Mail. For me, that is a more complicated process, as I will need to first import mail from a pre-Catalina Apple Mail backup, and then go through the migration of those folders to Outlook, while checking for possible duplicates of e-mails in Apple Mail, since some small proportion of the pre-Catalina mails survived the upgrade.
I would be sure that you have backups (more than one) of your e-mails as they exist prior to the switch, just to have a fall back position, if needed.
Be aware that there are some tradeoffs with respect to iCloud accounts when going with Outlook, based upon my preliminary experience. This is not a full list, but some highlights:
1. Outlook's calendar on macOS does not fully support iCloud calendar. You can "subscribe" to the iCloud calendar URL, in a read-only mode, but cannot create new appointments. You would either need to stay with using Apple Calendar, or consider migrating all of your calendar data to Outlook and then using that moving forward.
2. Curiously, the iOS version of Outlook does appear to support iCloud calendar in read/write mode, which is different behavior than on macOS.
3. Similar issues with Apple's Contacts in terms of read/write access, so you may want to consider switching to Outlook's contact system.
4. Presuming that you are using an iOS device, consider moving to Outlook there as well, which is what I will do, if I make the move on my Mac.
5. There are some general behavioral differences, but if perhaps you are used to using Outlook for work, then you should be comfortable.
Hope that helps.
Thanks, Mark.
I have all my emails on servers and download them to POP accounts on my Mac. I can download them all from the server to Outlook. It won't, of course, download my 'sent' emails or arrange them into folders as they are in Mac Mail.
Seems like an awful lot of bother.
Why on earth can't Apple sort this out?
I wonder if Keith Edwards has got anywhere after his email to Tim Cook..?
@John:
I haven't heard anything else after the original follow-up, although to be fair the person I spoke with said I wouldn't get any more info unless engineering needed to contact me about the issue.
Very disappointing to see that this doesn't appear to be addressed in the latest beta.
Keith,
Would you be willing to reach back out to your contact person anyway, despite what he told you? It certainly couldn't hurt, and it would keep this on his radar screen. I would certainly appreciate it if you did that.
Also, I think I'm going to do exactly what you did - write to Tim Cook and express my serious concerns about this and why nothing has happened.
Frank
I wonder if anyone can answer me this. The only emails I *really* need to keep for legal/tax purposes are the last 5 yrs online receipts. If I lose them I'm fu**ed, if the tax authorities investigate me.
If I highlight them in Mail and drag them all into a folder and then drag that folder to an external hard drive, after upgrading to Catalina, will I be able to open those emails from the external hard drive?
@john That would be a great use for EagleFiler , as it will make sure that the full message content (all attachments) are saved and that the data doesn’t get damaged during long-term storage.
If you want to do it with Mail, the best way is to drag and drop the messages directly to a folder on your hard drive, which will save them as .eml files. Dragging them to another mailbox in Mail first is bad because (even not on Catalina) it can lose attachments (and delete them from the server). Make sure that you test-open the .eml files to check that all the attachments are there; Mail won’t warn you if they’re missing.
For reasons that I don’t understand, when using drag and drop, Mail will sometimes save the files in .emlx format instead .eml, which is not good. If that happens, try again, or drag the messages individually. I do not recommend using the Export Mailbox command, as it generates malformed mbox files.
Thanks, Michael.
Have since spoken to a guy at Apple about this - another Mac technician entirely unaware of any Catalina Mail problems. He told me that opening the emails from a separate folder will still require communication with the Mail app to open them. So if Mail's corrupted by the Catalina update then they won't open. His advice: stick with Mojave for the foreseeable future. Not ideal.
However, what I did find was that, after saving emails to the folder on my desktop, I could choose to open them in Microsoft Outlook. I guess this might work after a Catalina update.
@john The .eml files can be opened by Apple Mail as well as most other e-mail clients. I don’t think you have to worry about opening them with Catalina Mail because the messages are detached from its data store. It’s just viewing them as individual files. You can also view them in Finder using Quick Look. Of course, staying with Mojave is safe, too.
Hello Fellow Sufferers,
I have a failing MBP, so I finally ordered a 2020 MBP 13 in, which is arriving later this month, with Catalina on it. I didn't know about this whole Mail issue, sadly, and I hadn't upgraded since El Capitan. I read several places that I should upgrade on this laptop to Catalina before migrating to the new laptop. Again, sadly, I did. The latest version, 10.15.4.
I don't seem to have a problem with distorted or empty messages, at least that I've discovered so far. I have the missing 4 GBs of emails problem. I opened Mail after upgrading and it was empty. No accounts, no emails. With help from a friend, I discovered the V7 folder in Library/Mail and it looks like all my emails are there. I have a Time Machine backup from right before I upgraded. (I have a SuperDuper backup and a Backblaze one, too, lol, because bad things have happened many times between It Just Works Apple and me.
I set up new accounts just fine, with the few emails still on the servers. I realized that I could make folders in the On My Mac section and import emails into them from V7 folder. I had many folders in the On My Mac section. I haven't tried importing any of the POP accounts yet, active or inactive. What I don't understand is this: If I import stuff from V7, when I migrate to the new laptop, will it be migrating what's newly replicated and arranged in my Mail app, or will it be just replicating from that V7 file and I'll have two versions of everything on the new laptop? If I import a mailbox from V7 to my Mail app, should I delete the source folder for that mailbox that's in V7 because the system will be updating as soon as I make a mailbox and THAT copy in V7 is the one I want? I hope that made sense. Import an On My Mac folder called Games from V7 folder into a folder that I name "Previous Games" in Mail app. Delete the the old folder in V7 that I just imported so I don't end up with both folders on the new laptop later this month.
I guess I do the same with the mboxs for the current POP accounts? Or if you think all of these might be corrupted in some way, import everything from Time Machine. Of course, I would keep a copy of the original V7 folder that the upgrade created, just in case. Any thoughts or warnings? I'm more tech-savvy than the average human, but much, much less tech savvy than many of you, I'm sure. Thank you so much for any thoughts.
@Carolyn People usually talk about importing from V6. V7 is the Catalina folder, so it normally doesn't make sense to import from the folder that’s already in use. I haven’t seen that before. That sounds more like a case where the Envelope Index needs to be rebuilt. And that you may eventually get duplicates (the next time a new database is created) if you’ve now created two copies of each mailbox in V7. Personally, I would start over with an empty V7 and import everything. When you migrate to a new Mac, nothing should change if you already have a V7. But, yes, keep copies of everything (and the backups) just in case.
@Robert Using Time Machine to backup IMAP content has not worked for a long time, since Mail may not be caching all of the data locally. You could either make a rule to move IMAP messages to an On My Mac mailbox so that they are fully saved locally or archive the messages using EagleFiler, which will ensure that the full message data has been downloaded. Either way, you’ll end up complete mail files that Time Machine can see. (Moving IMAP messages to an On My Mac mailbox manually—rather via a rule when they first come in—is not reliable and can cause parts of the messages to be permanently lost.)
Thanks, Michael. The Eaglefiler instructions make no sense to me whatsoever. I open a new Eaglefiler library, highlight a mailbox in Mail, press the f1 key and nothing happens. Or I try and drag the c.2,700 emails over and only one arrives.I just don't have time to drag every single email over that I need to save. I'm no technophobe but Eaglefiler has stumped me !
I have a somewhat similar situation to Carolyn above. I needed to upgrade my MacBook Air and I ordered a new MacBook Pro 13". I never upgraded my two macs(MacBook Air, Mac Mini) because someone told me about this bug back in October. I have kept my iOS devices updated to the latest versions with no apparent issues. The issue with the mail bug slipped my mind until today. It appears its still an issue and now I'm worried that I'll get this new mac and be unable to use anything related to mail on it because of this bug. I'm trying to get some idea of what I can expect if I do have an issue and how to best deal with it.
I've got three email accounts hooked up. One is a gmail account, one is an outlook.com exchange account and one is an IMAP account for a local ISP that I keep paying for email access from because so much is still tied to that address (I've had it for over 20 years) and I keep finding more every time I think I've migrated off. I don't have any pop3 accounts anymore (and email from the ones I had is already archived elsewhere). I'm a digital packrat and I don't delete legitimate (ie, not spam or fraud or something) email. I'm trying to get some idea of the scope of the problem. With those accounts, what can I expect?
I understand that apparently most people don't see a problem, but if I do, how can I check to see if its happening so I can try to mitigate any issues right away. If I do have an issue, will it affect the email stored on the server? I've seen references to mail lost due to the bug propagating to other computers because of mail sync, which makes me think it will. Are there any recommended mitigation/recovery steps if it does happen? Is it something that will keep happening or is it a "lose your mail once and you're done" thing I have two different sets of backups of both macs and I also think I'll just take both my current macs completely offline then power them off before I try to add my mail accounts to the new mac.
I'm starting to regret ordering that mac. Maybe I should have just bought a windows laptop instead.
Thanks!
@Ben As far as I know, the bug that deleted messages at rest (and propagated the deletion) has been fixed. The remaining bug is that (for some users) moving a message from one mailbox to another will delete it. Messages that had previously been moved by a rule seem to be at greater risk. So any old mail that is just sitting there should be fine. But messages that you drag and drop (or move in other ways) may be impacted at the time of the move. I don’t know of a 100% accurate way to test whether you are one of the unlucky users who are affected, but moving some test messages can give you an idea. Try moving messages that had previously been filed by a rule, try the different combinations (within the same account, between accounts, between server and local, etc.) and see what happens. For users who are affected, it nearly all messages moved in a certain way seem to disappear. If you’re using SpamSieve, it has a backup feature that keeps a copy of each message outside of Mail, so that you can recover any that are lost.
I was helping someone with this missing mail problem and in the end created a simple python script to help migrate from Apple Mail to Thunderbird. Hopefully this will help some readers out.
Start by creating a mail export ideally from your pre-Catalina version of Apple Mail (it /may/ work for an export with 'empty' mail from Catalina's Apple Mail, it's not clear if/how those exports are broken).
Make a copy of that export and operate on the copy.
The script is discussed [in an issue]( https://github.com/thundernest/import-export-tools-ng/issues/28 ) on the ImportExportTools NG plugin for Thunderbird. It can currently be [downloaded here]( https://raw.githubusercontent.com/obar/import-export-tools-ng/master/scripts/applembox2sbd.py ) with [usage instructions here]( https://github.com/obar/import-export-tools-ng#importing-from-apple-mail-mbox-format ) and soon will be found in [the ImportExportTools NG repository]( https://github.com/thundernest/import-export-tools-ng/ ) (instructions and script).
This shit is still happening. Mac Tech Support is useless. They don't get back to you. Engineers don't get back to them, etc. Rinse, cycle, repeat. NONE of this would have happened on Steve Jobs' watch. This is the worst probem I have ever had with an Apple product. I'm selling my stock. I've had it!
I discovered this problem immediately after using migration assistant.
Rather than recovering the old V6 folder from time machine, is it OK to just use target disk mode and drag the old V6 folder to my new mac, and then import to mail in Catalina?
Michael,sorry for the confusion. I migrated "out of the box" and was then prompted for the 10.15.4 update, which explains my problem.
Thanks for your help and for this resource!
@ Michael Tsai -- Is the "bug" to which you refer on 5/20,4:19 pm ("the bug that deleted messages at rest (and propagated the deletion) has been fixed") -- the same "bug" to which you refer in your sentence at 5/24, 1:21 pm ( "My understanding was that the migration bug is already fixed in macOS 10.15.4") ?
@HoustonMJ Yes. I’m not sure how many bugs there are in the code. But from the user perspective there’s the issue of messages that you’re not touching getting deleted during a migration or database rebuild, and the other issue of messages disappearing or going blank when you move them. I’m pretty sure that the first one really is fixed. The second one is not fixed, although I’ve seen some reports that it no longer happens in all of the same circumstances where it once did. So, perhaps, partially fixed.
@MichaelTsai -- Thank you for your clarification (5/24; 8:57 pm).
Can you set forth your current thinking as to courses of action that those of us who are using a pre-Catalina OSs can take as part of our moving to a new Apple computer with Catalina installed as the OS (and with the inability of using Mojave on the new computer)?
What specific actions, mail clients [in lieu of Catalina Apple Mail], email archive/filing apps [for example Eagle Filer], and otherwise, do you propose? Thank you.
@HoustonMJ
1. Before updating to Catalina, make a backup of your Mac.
2. Ideally, archive the messages to EagleFiler to make sure that you get full copies of the messages, including attachments. (Time Machine and other backup apps won’t warn you about the ones that aren’t cached locally.)
3. Update all the way to macOS 10.15.4 before launching Apple Mail on Catalina.
4. For most users, Apple Mail on 10.15.4 will be fine. But you should test to make sure that you are in this group.
5. If moving messages seems to work for you, and you’re using SpamSieve, enable the backup feature , just in case. You may also want to regularly archive messages to keep a copies (or simply move them) outside of Mail. (I do this, anyway, because it bogs down with lots of mail and in case something goes wrong on the mail server.)
6. If you don’t want to use Mail, I like GyazMail and MailMate. Other popular options are Airmail, Postbox, Outlook, and Thunderbird.
I have had this issue since 10.15.3 (I am now on 10.15.5) - applying rules now simply makes some messages COMPLETELY VANISH! I ended up deleting my white/trash lists (which were getting lengthy), and added one simple rule to move messages ending in stacksocial.com to my normal Inbox from the Junk folder. Once I applied it, half of the entries in the Junk folder vanished. I know because I searched for the one that I wanted to add to the rule if successful.
@john
Just to confirm your methodology.
You upgraded from Mojave to Catalina, and installed all Catalina updates to version 10.15.5, BEFORE opening Apple Mail at any point during the upgrade process?
Did you by chance check the size of your Apple Mail folder tree, before upgrading and then after, to see if there was a reduction in the size of the tree?
Or perhaps more coarsely, check for any changes in free disk space before opening Apple Mail and then after, to gauge how much mail content may have been lost?
Thanks.
Hi @marc.
Actually, I've found the email which I thought was missing! I had trashed it prior to the upgrade. So it was still in the 'bin'.
So: so far for me the upgrade has gone smoothly, with no emails lost.
Hi @john
Thanks for the clarification. I was getting very concerned that there were lingering upgrade issues, given your report.
Hope that it continues to go well for you.
I'm helping an old friend to find all her missing emails. It started when she bought a brand new MacBook for a few weeks ago, and the Apple Authorized reseller promised to help her with transferring all data from her old 2012 MacBook. And af course almost all messages was gone in Catalina. Around 20.000! What the reseller did was just migrating from the old OS in one big jump.
And now they claim that there were no emails from the beginning (!?)
I managed to take out the old hard disk and did a simple data recovery with a freeware. And found all the emails!
What's the safest way now to import the old mailboxes to Catalina Mail app?
I moved to Catalina, kicking and screaming as my 2018 Mac Mini running Mojave kept crashing when idle. Apple engineering told me to upgrade, implying that this was never going to be fixed.
Running Mojave I had about 10 local folders under "On My Mac".
After installing Catalina the folders are invisible. I currently have a ticket open with Apple on this.
A customer had Catalina 10.15.4 and two of three new email accounts hosted by the same provider added without problems. The third would not add. Apple of course blamed the email provider. On my machine, also running 10.15.4 I added all three without issue.
The client upgraded to 10.15.5 and was able to install the third account.
I have two other apps that install mail bundles and both bundles fail (SmallCubed MailSuite, and DEVONthink) with "…because the developer cannot be verified".
Apple has lost its way - sadly, the famous attention to detail has vanished.
Hi all,
A quick note to point out that through 10.15.6 beta 2, the version of Apple Mail included has not changed from 10.15.5 release.
It is still Version 13.4 (3608.80.23.2.2).
[…] stable, most disruptive-of-fluency macOS releases in recent memory. There are egregious issues of data loss in Apple Mail. Mail also simply won’t scroll — scroll! — certain HTML emails anymore. […]
Ugh! This Mail problem is a nightmare! Two months on the phone with Apple. I'm in a black hole with them. Clearly they are just spinning their wheels with me as they know this is a HUGE issue and they don't have a fix for it. I have disappearing mailboxes. reappearing deleted emails, cannot move emails with attachments between my import On My Mac and my iCloud side. If someone could recommend another email program that I could import my mesages into I would surely appreciate it! I miss Steve!
[…] Tsai, who has been tracking this issue, […]
I had a conversation today with someone in Apple's corporate office (I believe) about this whole matter.
This person followed up on my complaints that nothing was being done about this serious matter. They told me that, "I can assure you that we are aware of it"; however, they were non-committal about if or when it is going to be fixed. This individual learned from engineering that there was one major aspect of this that was already fixed (consistent with comments on this blog) but other aspects were not yet fixed. Reading between the lines, it sounded like they are having a hard time understanding the situations when this occurs, whether there are certain mail servers that are more likely to result in Mac Mail losing mail, etc. I think this person was trying to convey that they didn't know when or if this would be fixed, and that a fix wasn't imminent because they haven't isolated the problem. It's possible that it will eventually be fixed, and also possible that it won't.
So, while I wish I had received a different response, I appreciated the honesty (and I did think it was an honest response as I could tell that they had looked into it) and was at least glad to know not to keep waiting. I haven't yet upgraded to Catalina, waiting for a fix, and now I know to get myself a different e-mail program to use in preparation to upgrade.
I also gave feedback that it would be really nice to get better communication from Apple about these types of things, one aspect of which would be to issue "release notes" that provide more comprehensive information about what has been fixed.
So, there you have it. Hope that is helpful.
Looking at he comment trail the following.
I have issues with content disappearing from my email, moving emails from my Inbox to other mailboxes.
I run Catalina 10.15.5.
In the comment trail it looks like this is a known issue for quite a while. It seems like this issue is not solved in 10.15 and scheduled for 10.16. Is this a correct interpretation? And if 'yes' when is 10. 16 scheduled?
I am thinking of moving to Outlook since this bug is pretty annoying.
With a variety of problems in 10.5.5, I decided to try 10.5.6b2 because in all reality, the difference between an Apple beta and release since MacOS X is just the calendar date.
Anyway, I downloaded the beta enabler for 15.5.6 beta 2 and let it do its thing. I come back and I’m on macOS 11 AND Mail crashes at launch. https://twitter.com/cerniuk/status/1275774386193170433?s=21
To get mail to work, I had to remove the V8 folder under the Mail where it keeps its data (3GB) and it rebuilt it as V2 as if it was importing the data (from somewhere) but did it way way too fast. I never told Mail that the V8 folder was on the desktop. Hmmm.
In some ways macOS 11 beta is more stable than max is 10.5.5, in other ways not... but based upon this experience be ready to lose a lot of mail data when you migrate to macOS 11.
Been a bit since any traffic here.
Today, Apple released macOS Catalina 10.15.6.
The version of Mail that was in 10.15.5 was Version 13.4 (3608.80.23.2.2).
The version of Mail that is now in 10.15.6 is Version 13.4 (3608.120.23.2.1).
So, there has been a small incremental update in the app with this new macOS update. No documentation of any changes are included, so it is not clear what has actually changed.
[…] about fixes for problems users have experienced with Mail. However, the most recent comment in Michael Tsai’s long-running thread warning about data loss notes that Mail’s build number has changed, so it’s possible that Apple made some […]
Since the integrity of email is essential to me, I decided to move to MS Outlook…………
It puzzles me it takes so long for Apple to fix this mail problem…………I expected a higher standard
I decided a few weeks ago to give up on Mail and use Thunderbird. Not because of this problem, but because of many other Mail problems (example: I have never been able to learn how searching mail is supposed to work, and it fails on at least two thirds of my simple attempts, such as a From: address). I use a POP server, so may be immune from the disappearing mail problem.
The terrible aspect of this is that I will lose SpamSeive, which, as we all know, is as close to error free as we are likely to see for a very long time. I decided to hope that the Thunderbird Bayesian filter works well enough. I plan to use the tool by Gladwev to make the switch, one account at a time (extract, then import). Unless I learn of an alternative. (Since Michael mentioned it, I'll look at Eagle Filer.) I don't believe Thunderbird direct import is working.
Scrolling backward to read Michael's contributions, I read Frank's of June 18, 2020 2:51 PM.
Thank you Frank! I just bought a Mac Pro (late 2019) to replace my Mac Pro (late 2010). I will be forced onto Catalina. I have been talking to the folks at Apple second level Enterprise support, and they believe use of Migration Assistant will be painless.
After reading what Frank wrote (and having read other reports here) I have decided to let the new Mac sit empty until I convert all my mail to Thunderbird and see that working smoothly.
@Joaquin Were you not using POP, it would be possible to use another IMAP client to filter mail via SpamSieve while still using Thunderbird to read it. If you send me an e-mail, I can add you to a list to notify if/when there’s a better way to use it with Thunderbird. You could certainly use EagleFiler to extract the messages from Mail, and then have Thunderbird directly import from the EagleFiler archive. I would trust this over Mail’s own export, which has at times messed up the mbox format and (still) doesn’t warn about missing attachments (though that wouldn’t be an issue for POP.
Two days ago, I updated from High Sierra to Catalina. Mail worked flawlessly under Hs (apart from search, which was flakey at best), but in C, I find that moving messages out of Spam/Junk folders to inbox or any other folder deletes them irretrievably - they don't seem to exist anywhere any more. That's bloody annoying.
I've tried all the 'fixes' I can find on various sites, and none of them make any difference to the behaviour. All I can conclude is that Mail is broken.
I submitted a request to Apple months ago to look into this bug, but of course did not hear back. I'm responsible for many Mac models across my 2 businesses, not to mention at home. All are on 10.14.6 because I can't afford to lose important emails. I have iCloud, Gmail and Exchange accounts sync'd to many different mailboxes inside Mail and am frozen without a path forward. I would consider switching mail clients, but I use Daylite CRM in both of my businesses and it only links to emails from Mail. This is a terrible situation with no seeming path forward.
I upgraded from 10.14 to 10.15.6 Catalina on 7/29/20. Since then I have lost a couple of mails completely as I dragged them from one mailbox to another. They absolutely disappeared from my hard drive, as best I could tell with UNIX search tools. I was able to find one of them going to my backup TimeMachine drive, and by searching for a text phrase that I knew was unique the missing email. But if I wasn't paying attention, I would have lost it forever without even knowing what I was missing from my archive.
@Jack In Catalina, the message list in “classic” mode has fixed columns that can’t be configured or re-arranged. In Big Sur, it’s back to being a standard macOS table view, like in Mojave and earlier. This has nothing to do with the data loss bugs. I’m not aware of anyone on the inside saying those have been addressed. Commenter Jedidiah above says that he just lost some moved messages with macOS 10.15.6, and I have other reports of large numbers of Exchange messages at rest disappearing.
First I'd like to say thank you Michael for keeping comments open on this post. Your blog seems to be one of the few places were you can get some concrete information about the mysterious "moving-dissapearing email" bug.
I'm still running Mojave on two Macs because of a Fujitsu Scanner I use and because of the Mail.app bug. Fujitsu has recently released a 64bit version of ScanSnap Mangager, therefore theoretically I could now upgrade to Catalina. After being bitten by a nasty bug in Photos.app that deleted a couple of hundred pictures from my library in one of the early Mojave releases, I am now extremely cautious with macOS upgrades.
Here is my question. How easy is it to test or reproduce the email-moving-bug? I have a clean Catalina install on an external boot drive. I could configure my email account on this Catalina install and try moving a couple of emails and see what happens. Would this test be sufficient to know if my email is safe from the moving email bug?
@Dan From what I’ve heard, it’s easy to reproduce. For those affected by the bug, moving any message (or perhaps any message that had previously been moved by a rule) will make it disappear. If that succeeds, you are likely one of the lucky unaffected users (due unknown particulars of your Mail setup or server). I can’t promise that you’ll be safe, but I don’t recall hearing from anyone who was intermittently affected by the bug.
Intermittent fault with emails waiting to be sent, stuck in Apple Mail Outbox.
Eventually have to Force Quit Mail. Outbox nowhere to be seen on app reboot. Emails have evaporated. Not in drafts/sent/outbox empty.
Have rebuilt. Twice. Nothing.
Gone into ~Library etc. Nada.
Rebooted Macbook. Can send fresh emails as normal but still no sign of missing emails that were stuck in 'sending 5 items purgatory'.
Repaired permissions.
Checked mail settings - all OK.
Used Data recovery - nothing.
Fault doesn't happen often enough to be constant, consequently I've no way of reproducing it, it's so random. But it always seems to be on a complicated (not necessarily large) email that's taken me a while to pull together. 🙄
Two hours this evening on phone to Apple Support - eventually senior advisor. Nothing.
Also, any open drafts I'm working on when eg system crashes, often reappear after reboot as just a blank email. Empty 'To' field, empty 'Subject', zero content. So I don't even know what it is I've lost in order to try and recreate it.
2018 Macbook
1TB data
Plenty of free storage
32GB RAM
Running Catalina - 10.15.6 (19G73)
Any ideas? Before I commit violence...
I recently wrote a mail to federighi@apple.com , complaining about the state of this issue, and referencing this very page. I then received a reply by "craig" (or whoever answers this mail account at Apple) asking: "Are you personally experiencing this issue?", to which I replied that I didn't as I was still too afraid to upgrade to Catalina because of the comments on the Internet about this very issue.
So it might be worth it for anyone who _is_ actually experiencing the issue (and can maybe even reproduce it somewhat reliably) to write a short mail to federighi@apple.com stating this.
If you don't receive a reply in a couple of days, you might also try writing to tcook@apple.com instead, which is what I initially did when I first asked Apple about the state of this issue back in January when there was no fix in 10.15.2, to which I got a reply from the federighi@apple.com address, asking me which reports I was referring to. After I referred them to this very page, I didn't hear back anymore in January. So no actual idea, but maybe this triggered the fix for the first problem in 10.15.3 …
Missing (corrupted) imported Mail Messages ... continues in 10.15.6
Due to persistent kernel panics (10.15.6), I did a clean install. This included using CarbonCopyCloner (5.1.21) to move the Mail folder from a Time Machine (HFS+) backup disk to the iMac's (2015 I7 w 32GB) external 2TB boot disk (APFS). Then I ran CCC a second time to confirm the integrity of the copied files/folder. It passed without any apparent errors.
Then, using the Mail import function, I imported the >150,000 messages from the new Mail folder copy on the SSD.
The result: hundreds of messages devoid of text and attachments. But they all had the correct elmx file name.
Significantly, however: the corrupted Mail problem did not result from the Mail import function. Rather, the messages were clearly corrupted during the original copy process from the HFS+ backup to the APFS SSD target disk.
That is, the Mail folder copy, created with CCC contained corrupted Mail message documents/files. But the original Mail folder/directory still on the Time Machine disk did not.
How do I know? The corruption destroyed each message's contents but not the file names. So I could do a file by file comparison.
The corrupted files appear to be those that had some form of attachment: image, pdf, pages document. The corruption, from what I could tell, did not occur to messages comprised only of text.
Next I did a manual drag/drop copy of the same Mail folder from the Time Machine backup to the new SSD startup disk.
Good grief: no corrupted files whatsoever!
To draw some preliminary conclusions I'd want to repeat the processes several times to see if the results are consistent.
But that would be too time consuming for me just now.
Questions remain. Does CCC run a fundamentally different copy routine than manual drag/drop entails? Is there an issue related to moving files across the HFS+ - APFS divide, et cetera. Why does the problem only occur, from the few corrupted messages I checked, to those messages that contain an attachment of some sort - a linking problem/corruption?
Again I do not have the time to trouble shoot this fully ...
Thanks to everyone for contributing experiences and recommendations. According to Michael, the mass deletion of thousands of e-mails upon migration isn't so much an issue with the current version of MacOS Catalina. But for some, moving messages between folders can cause a disappearance. Most recent posts are about IMAP mail, not surprisingly. I am using POP3 with many rules for sorting incoming mail. So obviously, moving happens. Any current information on the risk of moving a POP message to another folder? Thanks.
I talked to an Apple Business Services rep yesterday and expressed my frustration over this bug. I told him I have 2 companies running Mojave and I'm afraid to upgrade all of our computers. He told me he thought the bug was fixed in Catalina, but I am also supposed to hear back from their systems engineer. We'll see... I just can't believe this has been going on so long as it is a terrible bug.
I have been avoiding Catalina because of this (there are other problems, but they might be able to be worked around). I don't even need the bug to be fixed immediately; what is causing it? Is there any way to know whether you will be affected? If there is some way to know who it affects and who it doesn't, then at least you can determine if you can upgrade or not. Right now, even if it only ever affected 10 people in the entire world, without *knowing* what causes it, I have to assume it will also bite me...
Michael Tsai : Your comment yesterday has me a bit confused so I am hoping that you could clarify things for me. Reading through all of the comments it seems that there were two initial bugs in the Mail app in Catalina:
1)Upgrading from Mojave resulting in losing many- if not all emails
2)Moving messages between Mailboxes resulted in the emails being empty and losing attachments
The first one seems to have been fixed (from earlier reports) so I will focus on the second issue.
My confusion is when the bug is an issue when moving messages around and whether it affects both IMAP and POP users.
Can you please answer the following questions from what you know?
1Are emails lost/corrupted when one moves messages between folders that reside on an IMAP server?
2)Are emails lost/corrupted when one moves messages from an IMAP server to a local(on the mac) folder/mailbox?
3)Are emails lost/corrupted when one moves messages that exist in a POP3 Inbox to other local(on the mac) folders/mailboxes - either manually or via defined rules?
My main email account is POP3 and I have rules that automatically move many messages into local folders/mailboxes on my Mac as they are downloaded into my Inbox. I also manually move many messages around locally on my Mac (from my Inbox as well as folder/mailbox to folder/mailbox.
I have been holding off on upgrading to Catalina because I thought that this bug affected both IMAP and POP3, but if it is only IMAP then I should be in good shape and could move ahead with the upgrade.
Thanks for your help and advice!
This may be a dumb question, but here goes...
I recently made the mistake of upgrading to Catalina, and am now at 10.15.6. For some time prior to that I had been using rules to archive my POP3 mail in 2 mailboxes On My Mac, on an older Mac running 10.13.6. There were about 4,500 messages in one of these and only about 1,500 in the other one. After I set up my newer Mac about a month ago and upgraded to Catalina, I discovered that all messages prior to July 1, 2020, were missing from both mailboxes on the newer Mac. I suspect this is not an indexing problem, but the messages are really gone. Fortunately, they are still complete on the old Mac, and most had been archived in MailSteward, and the MailSteward databases have been successfully copied to the new Mac. So, the question: if I deleted v7 from the Mail Data folder on the newer Mac, and replaced it with v5 from the older Mac, would Catalina recognize it and in effect, restore the missing messages to my 2 mailboxes?
This may be a dumb question, but here goes:
If I remove the v7 Mail Data folder from Catalina 10.15.6 and replace it with the v5 Mail Data folder from 10.13.6, will Catalina be able to read it and recover lost messages in it?
The back story is that I have been running 10.13.6 on another Mac, and had no email problems. I used outgoing and incoming rules to store most POP3 email into 2 mailboxes On My Mac folders; one had about 4,500, the other only about 1,500. Then I began using a newer Mac, and upgraded it to 10.15.6. I discovered that both of the above On My Mac mailboxes on the newer Mac only contained mail beginning in July 2020. I suspect this is not an indexing problem, and that the messages are really missing in Catalina. Fortunately I had archived all messages in those mailboxes with MailSteward, and have moved the MailSteward database over to the new Mac, so have recovered everything. But I still would like to get the missing mail back into Mail on my new Mac. I asked this question on Apple support communities and the only response I got was basically, "Try it and see," along with also deleting ~/Library/Containers/com.apple.mail and ~/Library/Preferences/com.apple.mail.plist.
I thought I would share some recent (9/1/2020) upgrade/migration experiences with the group. I few years ago I helped several friends transition from PCs to Macs and of course now I am their main source for tech support!
Migrated from a 15” Mid 2014 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6
Migrated from a 13” Early 2015 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6
Updated a 13” Early 2015 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6
Updated a 27” iMac Mid 2015 running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6
First two computers had users that had IMAP email accounts including Google and iCloud among others.
Second two computers are mine and I have a main POP3 account but also have Google and iCloud accounts that are IMAP.
Results:
In all cases emails were preserved in their entirety and moving messages between mailboxes/folders leaves the emails intact. The only issue noticed to date is that one of the users that migrated to Catalina from Mojave has noticed some indexing problems with Mail where emails are not being returned for specific searches even though they can be found individually.
I am not saying that based on my experience people should migrate/upgrade, I simply wanted to add my experience to the “collective.” It is disturbing that there seems to be no rhyme or reason as to who is affected by this particular bug.
Like many others, I'm still using Mojave because I cannot afford losing my mail history in IMAP folders. I will probably skip Catalina altogether and hope that Apple will be able to fix this bug in Big Sur. Does anybody have e-mail addresses of important people at Apple, so we can tell them about this nasty problem (again) and urge them to finally fix this long-standing bug?
I finally updated to Catalina on my trusty 2012 Retina MacBook Pro because I need to run Xcode 12. I happily exported all archived (local) mailboxes to EagleFiler and am currently trying out MailMate for active email. Did a clean install of Catalina on a new SSD (upgraded to 1 TB) and moved my data via Migration Assistant (unfortunately, SuperDuper! had failed to verify some internal partition).
Upgraded from 10.14.6 to 10.15.5 some weeks ago and am affected by this persisting bug since then.
Unfortunately, have to confirm that it still has not been resolved on the current 10.15.7 release.
If you are willing not to loose data from your IMAP Accounts, avoid Catalina by all means!
So this may be unrelated but I’ve had an issue for months where each time I opened Mail on Catalina, it would download thousands of emails from my dovecot server.
I could see it in activity manager using up roughly 100mb of bandwidth each time and though I don’t think I lost any emails, I figured out what was causing it.
A nested folder named “archive”. For whatever reason, Apple mail doesn’t like that, even though I had a separate folder named “Archive” at the top level. Anyways, renaming it “_archive” fixed it, and since this thread is getting a lot of attention, I wanted to add that to some repository of knowledge.
I have finally given up on Mail this week, I have wasted too much resolving issues. A couple of days ago Mail stopped receiving email from one of my accounts (worked fine on iOS devices), I tried rebuilding the mailbox and all obvious fixes. I have moved to Outlook, and must admit it was daunting at first, but after a couple of days am finding it a joy to use, feature rich and stable (and this from a Mac user since 1989)... It is disappointing that Apple do not have a stable and feature rich mail client, but perhaps it has given up the battle with Microsoft on this one...
Had Mail issue since probably 2 weeks ago. Could be because of the upgrade to 10.15.7 or O365 server side, not sure. Was all ok before. Originally it's my O365 work email stop receiving emails. Reopened the Mail client sometimes pull in the latest email, but not reliable at all.
Apple Support is USELESS. Asked me to collect a lot of information by their Capture_Data app and simply asked another rep called back and checked the connections doctor that i checked 100 times and called it a Server side issue. While Outlook and Spark getting emails fine. That O365 has around 70k emails over the years.
After many struggle nights, I even had to recover the whole mac from Time Machine backup because I lost some emails when working on another email address (this is somehow related to this article). Since I was working on the emails, I was moving an old email account from POP3 (which was set in POP3 when I started to use that for another work email) to IMAP. The export mailbox didn't seem to capture all the messages and I lost a lot of records (hence the need to recover from TM. TM seems to save the old message back, and the IMAP server seem to have email count correct, but now the IMAP can't download all the messages, it's stops around 18 months ago. While the server side has 8+ years worth of messages. This time both Outlook and Mail have the same issue.
It could be the messages got a bit corrupted during MacOS upgrade (as this article suspected) or server side issue, or TM didn't get all the messages backed up, or something else.
Still not sure if the original O365 issue was fixed or not as I'm still fixing all the headaches regarding emails.
Lastly, I just want to add Mac Mail still has the best search function. Outlook (while it's surprisingly stable and did not stop/break the download for no reason), its search is buggy in my exp. It can't find an email I got couple days ago (or it's still indexing...not sure). I hope Apple fixes Mail, as I am really ok with the "features" I get from it. Most of the Outlook features I can use other app.
Btw, another minor issue I found from Mail is the "Siri remind me this email" command many times link me to the wrong email. So the mailbox is probably corrupted in someway (or Mail is just broken). I've "Rebuild" the mailboxes TOO MANY times...
Can anyone comment on whether the "deletion upon moving a message between folders" issue remains in Catalina 10.15.7 or for that matter the most recent public beta of Mac OS Big Sur?
Thanks
I finally gave up on Catalina after one year, in particular, since mail started crashing every couple of minutes, in addition to mail with attachements disppearing when moving to my Mac. Installing Mojave was nontrivial and required a boot disk. The migration assistant did not work, but I finally got it done, including importing almost all of my mail in the V7 folder from Catalina. Using mail in Mojave is a pleasure - it was worth the effort.
I just found out about this bug the hard way. I lost an important email - can't find it anywhere. I am able to reproduce the bug as follows:
1. create a new mailbox and select location "On My Mac".
2. manually move (I used the pulldown menu "Move to >" command) a mail message (that you don't care to lose) from your iCloud inbox to the newly created mailbox on your Mac.
3. you can verify the move and view the message if you like.
4. close the message but keep it selected and move it again to an existing mailbox (for instance Junk - I used the pulldown menu with specific "Move To Junk" command).
5. Poof the message should now be gone.
I plan to report this to Apple. I believe the bug is triggered by first moving a message(s) to a mailbox local to your Mac hard drive and then moving it again to a cloud mailbox. Maybe there are other conditions at play as I have many mailboxes and accounts but this simple action reproduces the problem for me.
Thanks for maintaining this thread as it has helped me understand what happened in my case of the missing emails.
Finally, I have not tried any remedies other than to search for the emails using keywords specific to the headers/text. No results.
@johnnybrandom Thanks for writing out those steps. I’m seeing that same bug on 10.15.7. The destination mailbox briefly shows a number indicating that the message is there, and then it disappears. (I’m also hearing of messages disappearing when moved from server to local or between two servers, but I haven’t been able to reproduce that.)
@mjtsai - thanks for confirming the bug. I did report this to Apple using their Feedback Assistant yesterday. I've had a mixed bag with reporting bugs using Feedback Assistant - one previous report I filed did get a response from Apple engineers and a second did not. Hopefully this one will get some attention. Thanks again.
Just purchased a new MBP with Catalina. Everything was working fine. Woke up this morning and 3 IMAP accounts in MacMail were empty.... Went to the web host server and they're empty there now too. I literally just spent weeks directing all of my websites/service providers to these email addresses so I can have one easy place to look. Now they're all gone. I didn't move anything, sync anything, or do anything different. Last night at 10PM I got emails into those accounts, this morning everything is gone. I have an important email I need to respond to that came in late last night.... Any suggestions?
@johnnybrandom: If you do not receive a response through the Feedback Assistant, you might also try writing to